ckanext-subscribe

CKAN extension that allows users to subscribe to dataset/organization/group updates WITHOUT requiring them to login.

This feature is complementary to CKAN's existing "Follow" feature, which allows logged in users to subscribe to get update emails. Log-in can be a barrier to casual interest in say a handful of datasets. Generating and storing a password is a burden on the user, and for casual use just using temporary email links, as in this extension, is more appropriate.

More screenshots: https://github.com/bellisk/ckanext-subscribe/tree/master/doc

Requirements

Compatibility with core CKAN versions:

CKAN version	Compatibility
2.6 and earlier	no
2.7	yes
2.8	yes
2.9	not yet

Installation

To install ckanext-subscribe:

1. Activate your CKAN virtual environment, for example:

   . /usr/lib/ckan/default/bin/activate
   

2. Install the ckanext-subscribe Python package into your virtual environment:

   pip install ckanext-subscribe
   

3. Add subscribe to the ckan.plugins setting in your CKAN configuration file (by default the config file is located at /etc/ckan/default/production.ini).

4. Make sure that ckan.site_url is set correctly in the [app:main] section of your CKAN configuration file. This is used to generate links in the bodies of the notification emails. For example:

   ckan.site_url = https://example.com
   

5. Make sure that smtp.mail_from is set correctly in the [app:main] section of your CKAN configuration file. This is the email address that CKAN's email notifications will appear to come from. For example:

   smtp.mail_from = [email protected]
   

   This is combined with your ckan.site_title to form the From: header of the email that are sent, for example:

   From: Sunnyville Open Data <[email protected]>
   

   If you would like to use an alternate reply address, such as a "no-reply" address, set smtp.reply_to in the [app:main] section of your CKAN configuration file. For example:

   smtp.reply_to = [email protected]
   

6. If you do not have an SMTP server running locally on the machine that hosts your CKAN instance, you can change the email-settings to send email via an external SMTP server. For example, these settings in the [app:main] section of your configuration file will send emails using a gmail account (not recommended for production websites!):

   smtp.server = smtp.gmail.com:587
   smtp.starttls = True
   smtp.user = [email protected]
   smtp.password = your_gmail_password
   smtp.mail_from = [email protected]
   

7. Initialize the subscribe tables in the database:

   paster --plugin=ckanext-subscribe subscribe initdb
   

8. Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:

   sudo service apache2 reload
   

9. You need to run the 'send-any-notifications' command regularly. You can see it running on the command-line:

   paster --plugin=ckanext-subscribe subscribe send-any-notifications -c /etc/ckan/default/production.ini
   

   However instead you'll probably want a cron job setup to run it every minute or so. We're going to edit the cron table. On a development machine, just do this for your user:

   crontab -e
   

   Or a production machine use the 'ckan' user, instead of checking for notifications on the command-line, create CRON job. To do so, edit the cron table with the following command (it may ask you to choose an editor):

   sudo crontab -e -u ckan
   

   Paste this line into your crontab, again replacing the paths to paster and the ini file with yours:

   # m h  dom mon dow   command
     * *  *   *   *     /usr/lib/ckan/default/bin/paster --plugin=ckanext-subscribe subscribe send-any-notifications --config=/etc/ckan/default/production.ini
   

   This particular example will check for notifications every minute.

   Also in this cron you will likely see it also running a paster command for /api/action/send_email_notifications. This is similar but separate functionality, that core CKAN uses to send emails to users that have created user accounts e.g. for the 'follower' functionality. There's more about this here: https://docs.ckan.org/en/2.8/maintaining/email-notifications.html

Config settings

# Email notifications older than this time period will not be sent.
# So, after a pause in the sending of emails, when it restarts, it will not
# notify about activity which is:
# * older than this period, for immediate subscriptions
# * older than this period + 1 day, for daily subscriptions
# * older than this period + 1 week, for weekly subscriptions
# Accepted formats: ‘2 days’, ‘14 days’, ‘4:35:00’ (hours, minutes, seconds),
#                  ‘7 days, 3:23:34’, etc.
# See also: https://docs.ckan.org/en/2.8/maintaining/configuration.html#ckan-email-notifications-since
# (optional, default: ‘2 days’)
ckan.email_notifications_since = 24:00:00

# The time that daily and weekly notification subscriptions are sent (UTC
# timezone)
ckanext.subscribe.daily_and_weekly_notification_time = 09:00

# The day of the week that weekly notification subscriptions are sent
ckanext.subscribe.weekly_notification_day = friday

*** reCAPTCHA implementation ***
Applying reCAPTCHA helps enhance the security of the dataset subscription form by preventing automated bots from submitting them.

To integrate reCAPTCHA, you must first register your domain at https://www.google.com/recaptcha and obtain private and public keys.
These keys, along with the reCAPTCHA API URL, need to be configured in your setup::

 ckanext.subscribe.recaptcha.privatekey = [Your_Private_Key]
 ckanext.subscribe.recaptcha.publickey = [Your_Public_Key]
 ckanext.subscribe.recaptcha.api_url = [Your_API_URL], e.g., https://www.google.com/recaptcha/api/siteverify

Additionally, ensure that the `apply_recaptcha` configuration is set to true in order to enable reCAPTCHA::

 ckanext.subscribe.apply_recaptcha = false

Troubleshooting

Notification emails not being sent

1. Check your cron schedule is working:

   tail -f /var/log/syslog | grep subscribe
   

   You should see messages every minute:

   Jan 10 15:24:01 ip-172-30-3-71 CRON[29231]: (ubuntu) CMD (/usr/lib/ckan/default/bin/paster --plugin=ckanext-subscribe subscribe run --config=/etc/ckan/default/production.ini)
   

2. Create a test activity for a dataset/group/org you are subscribed to:

   paster --plugin=ckanext-subscribe subscribe create-test-activity mydataset --config=/etc/ckan/default/production.ini
   

   The log of the cron-activated paster command itself is not currently stored anywhere, so it's best to test it on the commandline:

   paster --plugin=ckanext-subscribe subscribe send-any-notifications --config=/etc/ckan/default/production.ini
   

   You should see emails being sent to subscribers of that dataset:

   2020-01-06 16:30:40,591 DEBUG [ckanext.subscribe.notification] send_any_immediate_notifications
   2020-01-06 16:30:40,628 DEBUG [ckanext.subscribe.notification] sending 1 emails (immediate frequency)
   2020-01-06 16:30:42,116 INFO  [ckanext.subscribe.mailer] Sent email to [email protected]
   

3. Clean up all test activity afterwards:

   paster --plugin=ckanext-subscribe subscribe delete-test-activity --config=/etc/ckan/default/production.ini
   

NameError: global name 'Subscription' is not defined

You need to initialize the subscribe tables in the database. See 'Installation' section above to do this.

KeyError: "Action 'subscribe_signup' not found"

You need to enable the subscribe plugin in your CKAN config. See 'Installation' section above to do this.

ProgrammingError: (ProgrammingError) relation "subscription" does not exist

You're running the tests with --reset-db and this extension doesn't work with that. Instead, if you need to wipe the tables before running tests, do it this way:

sudo -u postgres psql ckan_test -c 'drop table if exists subscription; drop table if exists subscribe_login_code; drop table if exists subscribe;'

or simply:

sudo -u postgres dropdb ckan_test
sudo -u postgres createdb -O ckan_default ckan_test -E utf-8
paster --plugin=ckan db init -c ../ckan/test-core.ini

Developer installation

To install ckanext-subscribe for development, activate your CKAN virtualenv and do:

git clone https://github.com/bellisk/ckanext-subscribe.git
cd ckanext-subscribe
python setup.py develop
pip install -r dev-requirements.txt

Now continue Installation steps from step 3

Tests

To run the tests, do:

nosetests --nologcapture --with-pylons=test.ini

To run the tests and produce a coverage report, first make sure you have coverage installed in your virtualenv (pip install coverage) then run:

nosetests --nologcapture --with-pylons=test.ini --with-coverage --cover-package=ckanext.subscribe --cover-inclusive --cover-erase --cover-tests

Releasing a new version of ckanext-subscribe

ckanext-subscribe should be available on PyPI as https://pypi.org/project/ckanext-subscribe. To publish a new version to PyPI follow these steps:

1. Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.

2. Update the CHANGELOG.md

3. Make sure you have the latest version of necessary packages:

   pip install --upgrade setuptools wheel twine
   

4. Create a source and binary distributions of the new version:

   python setup.py sdist bdist_wheel && twine check dist/*
   

   Fix any errors you get.

5. Upload the source distribution to PyPI:

   twine upload dist/*
   

6. Commit any outstanding changes:

   git commit -a
   git push
   

7. Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

   git tag 0.0.1
   git push --tags