From ca887dfd6659a55bc1f3545ff83d20ca54e061f1 Mon Sep 17 00:00:00 2001 From: Dirk Hoffmann Date: Wed, 14 Apr 2021 14:56:00 +0200 Subject: [PATCH] Update from upstream (#5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Let cloners know about multiple clones And bypass VC cloner availability check after the 1st clone (useful for e.g. Zoom) * Patch plugin version suffix in setup.cfg * Stop setting unnecessary click option The warning is only triggered in Python 2 with unicode_literals * Update dev docs for Python 3 * Show timetable filter in meeting-like conferences * Rename copyright.yml to headers.yml The old filename broke GitHub's license detection * Move package metadata from setup.py to setup.cfg Except for some dynamic parts... * Pin versions of dev dependencies * Expose dev requirements via [dev] extra * Sort requirements.dev.txt entries * Move test dependencies to requirements.dev.txt And get rid of unused pojson dependency * Pin minimum versions of pytz and certifi * We no longer have *.tpl files * Do not cache JS i18n data for invalid locales Otherwise each request for an invalid locale will create a cache file. Security scanners and similar tools may request a lot of invalid locales, and thus create a large amount of small but useless files in the cache folder (which are eventually cleaned up but not creating them to begin with is clearly better). * Fix downloading abstract/paper file packages The temporary file was being deleted too early when using xsendfile * Fix review_editable_revision outside request ctx * Update Chinese translation :cn: :dragon: * Add missing PR reference to changelog * Update French translation :fr: :baguette_bread: * Fix changelog wording * Release 2.3.3 * Bump version to 2.3.4-dev * Call metadata_postprocess for category event export When exporting all events from a category extending the metadata may also be useful, e.g. to add video conference join links. * Update url_for to default _external to False Flask's url_for _external defaults to True if there is no request context. * Make legacy event redirects routing-agnostic When changing the routing rules to require numeric event ids, we can no longer get leading-zero or non-numeric event ids from view_args * Always consider leading zeroes in event ids legacy Otherwise we can access a non-legacy event both with and without a leading zero, which is messy because there should be exactly one canonical URL * Replace confId with event_id everywhere Also make it numeric in the Flask routing rules * Update Jinja2 The previous version will soon receive a CVE which is not relevant for us, but let's make tools that only look at the version (e.g. GitHub's security notifications) happy. * Fix conference VC page if the VC plugin is missing * Fix deleting events with a VC room without the plugin * Update bleach to 3.3.0 The security issue does not affect our usage, but let's avoid warnings that may scare people... closes #4770 * Remove non-redis cache backends * Scope cache based on the indico version Like this we never risk errors due to cached data being from a previous version. * Add scoped caching layer on top of flask-caching Also improve the flask-caching behavior to not fail noisily in case redis is unreachable and support returning default values in case of cache misses. * Replace GenericCache with new scoped_cache * Remove GenericCache * Be more forgiving with invalid locales And warn if the configured default locale is invalid * Move indico.util.struct.* to indico.util.* * Add ID-1 page size for badge printing * Dashboard ical: Only query data that's needed * Fix eslint issues in Export.js * Hide semi-broken persistent ical links for contribs - They pointed to the event ical (without any contribution details) - When enabling (persistent) API keys from that widget, the URLs generated pointed to a legacy contribution-only ical api which 503s since v1.9.x - Persistent calendar links for a single contribution are not particularly useful * Fix event creation * Fix error in survey results - can't multiply strings with floats - dict_keys object can't be jsonified * Replace OrderedDicts with regular dicts (#4778) Regular dicts are sorted as of Python 3.7, so there is no need to use OrderedDict anymore. * Fix abstract submission `_.pluck()` was behaving very weirdly, and was supposedly removed in our lodash version anyway... * eslint person_link_widget.js * Allow timedelta for cache expiry * Use sane defaults in create_application fixture * Allow rejecting registrations with a reason (#4769) * Update *.pot files * cache: Fix timedelta conversion logic * cache: Fail noisily while in debug mode * Add temporary coverage file to gitignore * Use redis cache backend when running tests Using different backends in test and dev/prod is likely to result in errors that happen during regular usage but not in tests - and the fact that some tests actually contained bugs but passed nonetheless, proves this. * Support attaching iCal files to reminder/reg mails (#4780) * Fix dump_url_map That script initializes the app with testing=True, but we aren't inside pytest so we have no redis to connect to. * Fix dict ducktyping In Python 2 we used to check for iteritems+get which was safe, but items+get exists for quite a few motels, like the Survey one, and thus broke the ducktyping there and thus resulted in weird errors. * Avoid querying active registrations if not needed * Use 3.9 for RTD build * Remove unused schema instance * Remove weird session fallback in /api/user/ It was never used, and the idea of using the session user if there is any error with the oauth token (missing, invalid, whatever) is extremely obscure and prone to errors. * Add unit test for the complete oauth flow * Use Authlib for OAuth provider The implicit flow has been replaced with the more modern and in case of mobile apps more secure code+pkce flow. * Add token introspection endpoint * Disallow 'plain' PKCE code challenge It's less secure and we do not really care about clients that cannot create a SHA256 hash... * Add well-known oauth2 metadata endpoint * Allow CORS on token endpoint The PKCE flow can be used in browsers, so they should be able to exchange the code for a token via CORS * Improve error handling during authorization * Return more standard oauth errors * Remove obsolete oauth errors endpoint * Move oauth provider logic to core & split modules * Fix test_client with no webpack manifest If a test results in a templte being rendered this usually looks up some webpack bundles which we don't care about during tests and don't have during CI. * Add revocation endpoint * Apply app scope restrictions to existing tokens * Rename default_scopes to allowed_scopes The old name was just poor naming on the flask-oauthlib side, and never had anything to do with default scopes. * Support different tokens for different scopes Also refactor how user app authorizations are stored; instead of checking for a token, we now have a separate model holind the authorization and granted scopes, so if an app revokes its own token, the scopes will remain authorized in any future authorization flow the user does. * Remove unused property * Remove TODO comment about revoking tokens Deleting is fine, our tokens are unique so there is basically no risk of regenerating the same token that has been revoked before. * Fix error when jsonifying certain oauth errors * Add tests for insecure oauth logic * Stop expunging user from db session in tests Not sure why it now works without that ugly hack... * Add some app/user link assertions to the tests * Display metadata URL and add clipboard copy links * Remove unused authlib method upstream removed it as well * Make PKCE configurable Apps that are not targeting SPA/native apps do not need to allow flows without a client secret. * Test token reuse * Store OAuth tokens as SHA-256 hashes While not passwords, they are sensitive and there is no need to store plaintext versions of them. * Support implicit grant for the checkin app This also adds support for passing the token insecurely in the query string, but also ONLY for that app. Like this we are not under any pressure to update the app, and we can simply do it at some point after a proper Indico v3 release. * Add changelog entry for oauth * Fix merging users with oauth app auths * Use cascading FKs in oauth tables * Remove flower integration * ci: pin ubuntu version ubuntu-latest is now ubuntu 20 which doesn't work well with python 2 * ci: cache wheels + generate envs in setup This avoids weird random breakage with cached virtualenvs and is similar to what we already do for plugins * Remove deprecated registration form tab related css * Reduce label scope in the core css * Add primary_email_changed signal (#4802) Co-authored-by: Adrian Moennich * Improve testing behavior - database: prevent rollbacks - test client: run each request in a separate app context - add `make_test_client` fixture to create new clients * Unify current-request-user logic * Better handle auth fails during exception handling In that case the original exception is likely more relevant so we avoid re-raising the authentication error but rather silently discard the user that failed to authenticate. * Remove legacy url signing as much as possible We keep accepting the old links for the dashboard ical feed but no longer generate such URLs. * Simplify data in user_token We do not need to encode the data in the token, a simple signature is actually enough since the user id is prepended to the signature part of the token Also use sha256 instead of sha1 for the HMAC signature. While SHA1 is still fine for HMAC purposes, using a stronger algorithm doesn't have any disadvantages and the tokens are still reasonably short. * Remove leading whitespace in registrants export Happened if someone had no title * Handle oauth tokens in query strings To be removed when cc2cab2860 gets reverted (or earlier if we can ditch tokens-in-querystring before the implicit flow itself) * Use new user logic in oauth registrants API The CSRF checks no longer need to be disabled manually because any oauth-authenticated request is no longer subject to CSRF checks. * Do not register user picture url without user id We always put a user id in that url, and accessing it without one actually failed with a KeyError * Use new user logic in user info api * Remove override_request_user Looks like we don't need it in the end... * Preserve oauth errors if a bearer token is used The oauth RFCs actually specify statuscodes, so replacing everything with 400 is ugly * JSON requests are not likely seen by users * Make session user APIs clearer * Add changelog for session.user logic change * Correct some missing whitespace in templates * Fix URL signing when not logged in * Improve error handling in url signing * Use token-based urls for persistent ical links (#4801) Co-authored-by: Adrian Moennich * Fix registration form management * Exclude some user agents from sso redirection * Add bulk export for editing data to JSON * UI: add horizontal scroll to body Provide scrollbar if content overflows. * Allow Editing team to see editables of unpublished contribs * Update indico-sui-theme * Remove obsolete contextlib2 dependency ExitStack is part of contextlib in Python 3 * Use pip-tools to manage requirements.txt * Update Python dependencies * Use new flask-caching cache factory * Update to Flask-Multipass without open redirects * Fix open redirects in signup/logout * Enforce the BASE_URL to match the actualr equest - Always set `SERVER_NAME` and `APPLICATION_ROOT` - Improve the handling of requests where this does not match (fail with a meaningful error) * Update French translation :fr: :baguette_bread: * Redirect to canonical URL in the web server * Release 2.3.4 * Bump version to 2.3.5-dev * Add 2.3.5 changelog stub * Fix react warning about duplicate key * Add better password security checks * Check password security during login * Add autocomplete="{new,current}-password" metadata * Add Flask-Limiter for rate limiting * Add rate limiting for failed login attempts * Fix incorrect cache/limiter initialization This broke building assets * Only disable jest tests instead of all JS tests We try building the assets which catches errors such as the url map extraction script being broken * Fix error on contribution author page * Fix static program page having header+footer * Don't show deleted long-lasting events in calendar * Allow filtering participant roles by unregistered * Fix custom role members not showing as registered (unless they had another event role) * Add bulk export for paper peer reviewing data to JSON * Fix oversized badges on buttons fixes one of the issues in #4821 * Fix i-button-label losing boldness on hover fixes one of the issues in #4821 * Improve iCal metadata (#4820) * Do not emit empty location property (#4785) * Include contact data in iCalendar (#4786) * Include event logo url in public events (#4787) * Fix icalendar property name * Update robots.txt * Remove legacy xmlGateway endpoint * Fix nonsense in the docs * Allow unstaging users by clicking them again * Allow unstaging users from staged users list * Flash the staged user count label when it appears Like this people are more likely to notice it * Editing: Only show menu links the user can access * Fix weeks view theme * Check for duplicate users during registration import * Fix "inheriting acls" info - show the message about inherited access again - properly show each acl type when viewing the list - remove obsolete code * Remove weird `is_*` attrs on principal models Using the enum is much cleaner, and they were only used in few places (which are now using the principal_type). * Allow groups/roles as authorized abstract submitters * Use StrictUndefined for top-level objects * Be explicit about undefined callers * Be a bit more strict with undefined attrs/items * Add tests for our undefined logic * Celery: execute tasks immediately while testing * Fix error when notifying new paper reviewing roles * Use new sentry-python SDK instead of raven * Fix user id not being logged with empty session * Fix typo in changelog * Disallow .bin for "all formats" http api endpoints It's specifically meant for the file apis, and for anything else it attempts to send the dict with the original data, which fails with an ugly error. * Fix error with flask-limiter and partials We do not use request-level rate limiting at the moment, so just disabling the before-request check avoids this bug which broke endpoints using `RHSimple.wrap_function`. Related upstream issue: alisaifee/flask-limiter#124 * Update lxml and cryptography Both updates contain security improvements * Fix incorrect i18n formatting * Fix Undefined error in registrant list * Redirect meeting elements to slug-based URL * Fix error when marking registration as paid * Update event QR code for latest checkin app (#4844) Related checkin app PR: indico/indico-checkin#10 * Remove legacy oauth features for checkin app (#4846) * Revert "Handle oauth tokens in query strings" This reverts commit e1e8dbbdcff60c10807752bfe70199def51ee7bc. Also fixes the unit test since the /api/user/ endpoint returns null if not authenticated instead of failing with 40x * Revert "Support implicit grant for the checkin app" This reverts commit cc2cab2860d0d54b4528861365a34aeabfa9fef1. * Fix another UndefinedError * Fix subcontribution count check * Fix another UndefinedError * Add download option to ical export popups (#4850) * Fix UndefinedError * verbose_iterator: Support printing total duration * verbose_iterator: Fix error in case of no items * Update urllib3 (dependabot alert) * Remove invalid property * Remove horizontal scrollbar in Firefox * Remove unused get_nested_attached_items util indico/indico-plugins#109 removed the need for it, and in any case it would be something for the plugin if ever needed again. * Fix timetable details with explicit session access When a user has no access to the event but to a session, they should be able to view the timetable entry details for the contributions within that session. * Fix contribution access with explicit-only access ie a user who cannot access the event itself but is on the ACL for a contribution * Hide registration menu item if user has no access When an event is protected and the user has only access to a specific session/contribution inside, it makes no sense to show the registration menu unless the registrations are actually available to people with no event access. * Do not show empty conference menu We still have the blank area, but avoid showing just the border of the menu list. * Fix typo * Create SECURITY.md * Avoid inconsistent state when file deletion fails This can happen if there's a race condition between something claiming the file (and storing a reference to it in FK) + marking it as claimed and the file being deleted. Now we still fail in such a case, but no longer keep a file in the database that's gone from storage. * Fix reinitializing FileManager during "make changes" The tags are not guaranteed to be sorted, so sometimes the re-render caused by `setLoading(true)` resulted in the final-form's initialValues changing and thus reinitializing the form, triggering the file manager's logic to reset it as well and delete any pending files. Since this happened in parallel with the submission of the reviewing action, it at least resulted in the deleting request failing, but often also in a race between the deletion request and it checking whether the file has been claimed and the review claiming it, which resulted in the file being deleted from storage but not the database (fixed in a separate commit). * Add an option to keep Editing team anonymous * Add support for Polish translation :poland: * Add Polish translation :poland: * Update python deps; pin some trickier ones * Update to SQLAlchemy 1.4 * pytest: Only ignore a specific sqlalchemy warning * Fix some deprecation warnings * Cover `unknown` and mutability in webargs tests * Update to webargs 8 * Fix error when querying room statistics * Remove likely unnecessary joinedload This broke with SA 1.4 (sqlalchemy/sqlalchemy#6253) but doesn't seem to be necessary anyway. Also added some tests. They are kind of dumb (not testing the actual filtering logic) but they do cover the part that was broken. * Fix various Room Booking issues (#4861) * Make sure past bookings can't be cancelled * Show correct numbers when cancelling bookings * Fix timeline popup skidding * Fix fetchActiveBookings params * Ellipsize plugin version * Update JS deps within semver Pinned react-leaflet-draw to a specific version since newer versions within the 0.19.x line require react-leaflet v3... * Update package-lock.json * Update JS deps with semver-major changes * Update package-lock.json * Update package-lock.json to v2 (npm 7) * Update package-lock.json (npm upgrade) * Replace obsolete shortid package with nanoid * Fix eslint config The latest prettier plugin is no longer split into separate subconfigs * Enable legacy-peer-deps in npmrc enzyme and useAxios expect older react versions as peer deps and we don't care about this... * Add (n)pgettext support * Apply i18n to category event list date formats * Update *.pot files * Regenerate package-lock.json & fix ssh urls * `npm update` once again * Run latest pyupgrade * Use correct secure_filename fallback values * Be more verbose if plugin assets have not been built Co-authored-by: Pedro Ferreira Co-authored-by: Adrian Moennich Co-authored-by: Indico Team Co-authored-by: Pedro Lourenço Co-authored-by: Alejandro Avilés Co-authored-by: Ergys Dona Co-authored-by: Vasant Vohra Co-authored-by: Javier Ferrer Co-authored-by: Giorgio Pieretti Co-authored-by: Javier Ferrer Co-authored-by: Parth Shandilya --- .coveragerc | 1 - .eslintrc.yml | 20 +- .flake8 | 3 + .github/workflows/ci.yml | 174 +- .gitignore | 1 + .isort.cfg | 3 +- .npmrc | 1 + .readthedocs.yml | 4 +- .travis.yml | 39 - .watchmanconfig | 2 +- CHANGES.rst | 282 +- CODE_OF_CONDUCT.md | 4 +- CONTRIBUTING.md | 4 +- DEVELOPMENT.md | 9 + MANIFEST.in | 2 +- README.md | 4 +- SECURITY.md | 25 + __mocks__/axios.js | 2 +- __mocks__/react.js | 2 +- babel.config.js | 5 +- bin/maintenance/build-assets.py | 24 +- bin/maintenance/build-wheel.py | 34 +- bin/maintenance/dump_url_map.py | 15 +- bin/maintenance/make-release.py | 29 +- bin/maintenance/update_backrefs.py | 22 +- bin/maintenance/update_browsers.js | 2 +- bin/maintenance/update_header.py | 154 +- bin/utils/apiProxy.py | 12 +- bin/utils/create_module.py | 32 +- bin/utils/db_diff.py | 43 +- bin/utils/db_log.py | 31 +- bin/utils/room_occupancy.py | 13 +- bin/utils/storage_checksums.py | 12 +- conftest.py | 2 +- docs/source/api/oauth.rst | 14 +- docs/source/building/index.rst | 64 + docs/source/conf.py | 17 +- docs/source/config/settings.rst | 84 +- docs/source/exec_directive.py | 9 +- docs/source/http_api/common.rst | 1 + docs/source/http_api/exporters/user.rst | 36 +- docs/source/index.rst | 9 + docs/source/indico_uml_directive.py | 4 +- docs/source/installation/development.rst | 46 +- .../installation/production/centos/apache.rst | 6 +- .../installation/production/centos/nginx.rst | 6 +- .../installation/production/debian/apache.rst | 4 + .../installation/production/debian/nginx.rst | 4 + docs/source/installation/translations.rst | 33 +- docs/source/plugins/models.rst | 1 - headers.yml | 11 + indico/__init__.py | 12 +- indico/cli/cleanup.py | 8 +- indico/cli/core.py | 7 +- indico/cli/database.py | 30 +- indico/cli/devserver.py | 22 +- indico/cli/event.py | 17 +- indico/cli/i18n.py | 6 +- indico/cli/maintenance.py | 13 +- indico/cli/setup.py | 155 +- indico/cli/shell.py | 32 +- indico/cli/user.py | 43 +- indico/cli/util.py | 18 +- indico/cli/watchman.py | 10 +- indico/core/auth.py | 23 +- indico/core/cache.py | 232 +- indico/core/cache_test.py | 71 + indico/core/celery/__init__.py | 7 +- indico/core/celery/blueprint.py | 4 +- indico/core/celery/cli.py | 69 +- indico/core/celery/controllers.py | 4 +- indico/core/celery/core.py | 43 +- indico/core/celery/flower.py | 102 - .../core/celery/templates/celery_tasks.html | 9 - indico/core/celery/util.py | 21 +- indico/core/celery/views.py | 4 +- indico/core/config.py | 30 +- indico/core/db/__init__.py | 4 +- indico/core/db/sqlalchemy/__init__.py | 2 +- indico/core/db/sqlalchemy/attachments.py | 18 +- indico/core/db/sqlalchemy/colors.py | 12 +- indico/core/db/sqlalchemy/core.py | 30 +- indico/core/db/sqlalchemy/custom/__init__.py | 2 +- indico/core/db/sqlalchemy/custom/greatest.py | 2 +- indico/core/db/sqlalchemy/custom/int_enum.py | 19 +- .../core/db/sqlalchemy/custom/ip_network.py | 14 +- indico/core/db/sqlalchemy/custom/least.py | 2 +- indico/core/db/sqlalchemy/custom/natsort.py | 4 +- .../core/db/sqlalchemy/custom/static_array.py | 6 +- indico/core/db/sqlalchemy/custom/unaccent.py | 14 +- .../core/db/sqlalchemy/custom/utcdatetime.py | 2 +- indico/core/db/sqlalchemy/descriptions.py | 10 +- indico/core/db/sqlalchemy/links.py | 38 +- indico/core/db/sqlalchemy/locations.py | 34 +- indico/core/db/sqlalchemy/logging.py | 23 +- indico/core/db/sqlalchemy/migration.py | 15 +- indico/core/db/sqlalchemy/notes.py | 12 +- indico/core/db/sqlalchemy/principals.py | 98 +- indico/core/db/sqlalchemy/protection.py | 49 +- indico/core/db/sqlalchemy/review_comments.py | 4 +- indico/core/db/sqlalchemy/review_questions.py | 11 +- indico/core/db/sqlalchemy/review_ratings.py | 13 +- .../core/db/sqlalchemy/searchable_titles.py | 10 +- indico/core/db/sqlalchemy/util/management.py | 21 +- indico/core/db/sqlalchemy/util/models.py | 84 +- indico/core/db/sqlalchemy/util/models_test.py | 6 +- indico/core/db/sqlalchemy/util/queries.py | 34 +- indico/core/db/sqlalchemy/util/session.py | 6 +- indico/core/emails.py | 21 +- indico/core/errors.py | 15 +- indico/core/limiter.py | 78 + indico/core/logger.py | 140 +- indico/core/marshmallow.py | 29 +- indico/core/notifications.py | 36 +- indico/core/oauth/__init__.py | 39 + indico/core/oauth/endpoints.py | 48 + indico/core/oauth/grants.py | 62 + .../rpc/handlers.py => core/oauth/logger.py} | 9 +- .../fossils => core/oauth/models}/__init__.py | 0 indico/core/oauth/models/applications.py | 258 + .../oauth/models/applications_test.py | 12 +- indico/core/oauth/models/tokens.py | 136 + indico/core/oauth/models/tokens_test.py | 24 + indico/core/oauth/oauth2.py | 33 + indico/core/oauth/oauth2_test.py | 457 + indico/core/oauth/protector.py | 56 + indico/core/oauth/scopes.py | 18 + .../oauth/testing}/__init__.py | 0 indico/core/oauth/testing/fixtures.py | 58 + indico/core/oauth/util.py | 70 + indico/core/permissions.py | 73 +- indico/core/plugins/__init__.py | 81 +- indico/core/plugins/alembic/env.py | 17 +- indico/core/plugins/blueprint.py | 4 +- indico/core/plugins/controllers.py | 12 +- indico/core/plugins/templates/index.html | 4 +- indico/core/plugins/views.py | 4 +- indico/core/sentry.py | 95 + indico/core/settings/__init__.py | 4 +- indico/core/settings/converters.py | 21 +- indico/core/settings/models/base.py | 39 +- indico/core/settings/models/settings.py | 13 +- indico/core/settings/models/settings_test.py | 2 +- indico/core/settings/proxy.py | 77 +- indico/core/settings/proxy_test.py | 2 +- indico/core/settings/util.py | 20 +- indico/core/signals/__init__.py | 2 +- indico/core/signals/acl.py | 2 +- indico/core/signals/agreements.py | 2 +- indico/core/signals/attachments.py | 4 +- indico/core/signals/category.py | 4 +- indico/core/signals/core.py | 10 +- indico/core/signals/event/__init__.py | 2 +- indico/core/signals/event/abstracts.py | 2 +- indico/core/signals/event/contributions.py | 2 +- indico/core/signals/event/core.py | 5 +- indico/core/signals/event/designer.py | 6 +- indico/core/signals/event/notes.py | 2 +- indico/core/signals/event/persons.py | 4 +- indico/core/signals/event/registration.py | 15 +- indico/core/signals/event/timetable.py | 2 +- indico/core/signals/event_management.py | 2 +- indico/core/signals/menu.py | 3 +- indico/core/signals/plugin.py | 2 +- indico/core/signals/rb.py | 4 +- indico/core/signals/rh.py | 4 +- indico/core/signals/users.py | 16 +- indico/core/storage/__init__.py | 2 +- indico/core/storage/backend.py | 65 +- indico/core/storage/backend_test.py | 30 +- indico/core/storage/models.py | 52 +- indico/core/webpack.py | 6 +- indico/legacy/common/cache.py | 352 - indico/legacy/common/contribPacker.py | 56 - indico/legacy/common/output.py | 67 +- indico/legacy/common/utils.py | 79 +- indico/legacy/common/xmlGen.py | 16 +- indico/legacy/fossils/user.py | 72 - indico/legacy/pdfinterface/base.py | 110 +- indico/legacy/pdfinterface/conference.py | 290 +- indico/legacy/pdfinterface/latex.py | 68 +- indico/legacy/services/implementation/base.py | 54 - .../legacy/services/implementation/search.py | 79 - .../legacy/services/interface/rpc/__init__.py | 0 indico/legacy/services/interface/rpc/json.py | 23 - .../legacy/services/interface/rpc/process.py | 72 - indico/legacy/services/tools.py | 15 - indico/legacy/webinterface/pages/static.py | 6 +- indico/migrations/env.py | 17 +- ...38_2af245be72a6_review_questions_models.py | 10 +- ...72d63acba9_update_map_aspects_structure.py | 2 +- ...7c45c384d65_make_equipment_types_global.py | 4 +- ...c410be271df_make_room_attributes_global.py | 4 +- ..._cbe630695800_add_room_principals_table.py | 11 +- ...8e3b_migrate_event_labels_from_settings.py | 2 +- ...migrate_review_conditions_from_settings.py | 8 +- ...disallow_editing_permissions_for_groups.py | 2 +- ...3_1431_8d614ef75968_allow_mx_user_title.py | 62 + ...ntil_approved_regform_modification_mode.py | 30 + ..._2232_e787389ca868_add_rejection_reason.py | 26 + ...985db8ed12_add_attach_ical_to_reminders.py | 26 + ...254_add_attach_ical_to_registrationform.py | 26 + ...782de7970da_rename_oauth_default_scopes.py | 23 + ..._separate_authorized_scopes_from_tokens.py | 77 + ...c23c7_make_oauth_pkce_flow_configurable.py | 27 + ...914_d354278c6d95_store_tokens_as_hashes.py | 69 + ...cc7088914e7_use_cascading_fks_for_oauth.py | 35 + ...08_26806768cd3f_remove_flower_oauth_app.py | 37 + indico/modules/__init__.py | 2 +- indico/modules/admin/__init__.py | 4 +- indico/modules/admin/controllers/base.py | 6 +- indico/modules/admin/views.py | 4 +- indico/modules/announcement/__init__.py | 4 +- indico/modules/announcement/blueprint.py | 4 +- indico/modules/announcement/controllers.py | 4 +- indico/modules/announcement/forms.py | 4 +- indico/modules/announcement/views.py | 4 +- indico/modules/api/__init__.py | 8 +- indico/modules/api/blueprint.py | 13 +- indico/modules/api/controllers.py | 69 +- indico/modules/api/forms.py | 4 +- indico/modules/api/models/keys.py | 14 +- indico/modules/api/templates/_messages.html | 24 - .../modules/api/templates/user_profile.html | 2 +- indico/modules/api/views.py | 4 +- indico/modules/attachments/__init__.py | 15 +- indico/modules/attachments/api/hooks.py | 6 +- indico/modules/attachments/api/util.py | 8 +- indico/modules/attachments/blueprint.py | 33 +- indico/modules/attachments/client/js/index.js | 313 +- .../modules/attachments/client/js/legacy.js | 316 + .../modules/attachments/client/js/package.js | 60 + indico/modules/attachments/clone.py | 4 +- .../modules/attachments/controllers/compat.py | 13 +- .../attachments/controllers/display/base.py | 6 +- .../controllers/display/category.py | 4 +- .../attachments/controllers/display/event.py | 20 +- .../controllers/display/event_test.py | 132 + .../attachments/controllers/event_package.py | 82 +- .../controllers/management/base.py | 28 +- .../controllers/management/category.py | 4 +- .../controllers/management/event.py | 4 +- .../modules/attachments/controllers/util.py | 12 +- indico/modules/attachments/forms.py | 18 +- indico/modules/attachments/logging.py | 20 +- .../modules/attachments/models/attachments.py | 38 +- indico/modules/attachments/models/folders.py | 27 +- .../attachments/models/folders_test.py | 4 +- .../attachments/models/legacy_mapping.py | 17 +- .../modules/attachments/models/principals.py | 11 +- indico/modules/attachments/operations.py | 6 +- indico/modules/attachments/preview.py | 33 +- indico/modules/attachments/tasks.py | 26 + .../attachments/templates/_attachments.html | 4 +- .../attachments/templates/_display.html | 24 +- .../templates/generate_package.html | 14 +- indico/modules/attachments/util.py | 34 +- indico/modules/attachments/views.py | 4 +- indico/modules/auth/__init__.py | 39 +- indico/modules/auth/blueprint.py | 4 +- indico/modules/auth/controllers.py | 82 +- indico/modules/auth/forms.py | 39 +- indico/modules/auth/models/identities.py | 14 +- .../auth/models/registration_requests.py | 7 +- indico/modules/auth/providers.py | 12 +- indico/modules/auth/templates/accounts.html | 18 + .../emails/link_identity_verify_email.txt | 18 +- .../emails/register_verify_email.txt | 14 +- .../auth/templates/emails/reset_password.txt | 14 +- indico/modules/auth/templates/login_page.html | 8 + indico/modules/auth/util.py | 25 +- indico/modules/auth/views.py | 4 +- indico/modules/bootstrap/blueprint.py | 4 +- indico/modules/bootstrap/client/js/index.js | 2 +- indico/modules/bootstrap/controllers.py | 17 +- indico/modules/bootstrap/forms.py | 4 +- .../bootstrap/templates/flash_messages.html | 8 +- indico/modules/categories/__init__.py | 4 +- indico/modules/categories/blueprint.py | 12 +- indico/modules/categories/client/js/base.jsx | 34 + .../modules/categories/client/js/calendar.js | 2 +- .../js/components/CategoryStatistics.jsx | 2 +- .../components/CategoryStatistics.module.scss | 2 +- .../modules/categories/client/js/context.js | 2 +- .../modules/categories/client/js/display.js | 2 +- indico/modules/categories/client/js/index.js | 5 +- .../categories/client/js/management.js | 138 +- indico/modules/categories/compat.py | 8 +- .../modules/categories/controllers/admin.py | 4 +- indico/modules/categories/controllers/base.py | 6 +- .../modules/categories/controllers/display.py | 165 +- .../categories/controllers/management.py | 58 +- indico/modules/categories/controllers/util.py | 131 + indico/modules/categories/fields.py | 33 +- indico/modules/categories/forms.py | 28 +- indico/modules/categories/legacy.py | 6 +- indico/modules/categories/models/__init__.py | 4 +- .../modules/categories/models/categories.py | 49 +- .../categories/models/categories_test.py | 4 +- .../categories/models/legacy_mapping.py | 10 +- .../modules/categories/models/principals.py | 7 +- indico/modules/categories/models/roles.py | 15 +- indico/modules/categories/models/settings.py | 8 +- indico/modules/categories/operations.py | 4 +- indico/modules/categories/serialize.py | 58 +- indico/modules/categories/settings.py | 8 +- indico/modules/categories/tasks.py | 8 +- .../templates/category_export_ical.html | 17 - .../categories/templates/display/base.html | 16 +- .../categories/templates/display/sidebar.html | 2 +- .../management/_create_category_button.html | 8 + .../templates/management/_events_list.html | 4 + .../categories/templates/management/base.html | 8 +- .../templates/management/content.html | 53 +- .../templates/management/settings.html | 2 + indico/modules/categories/util.py | 23 +- indico/modules/categories/views.py | 10 +- indico/modules/cephalopod/__init__.py | 4 +- indico/modules/cephalopod/blueprint.py | 4 +- indico/modules/cephalopod/client/js/index.js | 2 +- indico/modules/cephalopod/controllers.py | 12 +- indico/modules/cephalopod/forms.py | 4 +- indico/modules/cephalopod/util.py | 16 +- indico/modules/cephalopod/views.py | 4 +- indico/modules/core/__init__.py | 4 +- indico/modules/core/blueprint.py | 6 +- .../modules/core/client/js/impersonation.js | 51 +- indico/modules/core/client/js/index.js | 2 +- indico/modules/core/client/js/session_bar.js | 2 +- indico/modules/core/client/js/top_bars.js | 2 +- indico/modules/core/controllers.py | 113 +- indico/modules/core/forms.py | 4 +- indico/modules/core/settings.py | 4 +- .../core/templates/admin/settings.html | 10 - indico/modules/core/views.py | 4 +- indico/modules/designer/__init__.py | 28 +- indico/modules/designer/blueprint.py | 6 +- indico/modules/designer/client/js/index.js | 2 +- indico/modules/designer/controllers.py | 18 +- indico/modules/designer/forms.py | 4 +- indico/modules/designer/models/images.py | 11 +- indico/modules/designer/models/templates.py | 11 +- indico/modules/designer/operations.py | 4 +- indico/modules/designer/pdf.py | 10 +- indico/modules/designer/placeholders.py | 13 +- indico/modules/designer/templates/list.html | 2 +- .../modules/designer/templates/template.html | 6 +- indico/modules/designer/util.py | 23 +- indico/modules/designer/views.py | 4 +- indico/modules/events/__init__.py | 54 +- indico/modules/events/abstracts/__init__.py | 6 +- indico/modules/events/abstracts/blueprint.py | 8 +- .../events/abstracts/client/js/boa.jsx | 18 +- .../events/abstracts/client/js/index.js | 2 +- indico/modules/events/abstracts/clone.py | 12 +- indico/modules/events/abstracts/compat.py | 8 +- .../events/abstracts/controllers/abstract.py | 14 +- .../abstracts/controllers/abstract_list.py | 22 +- .../events/abstracts/controllers/base.py | 14 +- .../events/abstracts/controllers/boa.py | 10 +- .../events/abstracts/controllers/common.py | 26 +- .../events/abstracts/controllers/display.py | 10 +- .../abstracts/controllers/email_templates.py | 6 +- .../abstracts/controllers/management.py | 18 +- .../events/abstracts/controllers/reviewing.py | 20 +- indico/modules/events/abstracts/fields.py | 36 +- indico/modules/events/abstracts/forms.py | 104 +- indico/modules/events/abstracts/lists.py | 92 +- .../events/abstracts/models/abstracts.py | 23 +- .../abstracts/models/call_for_abstracts.py | 12 +- .../events/abstracts/models/comments.py | 7 +- .../events/abstracts/models/email_logs.py | 9 +- .../abstracts/models/email_templates.py | 13 +- .../modules/events/abstracts/models/fields.py | 9 +- .../modules/events/abstracts/models/files.py | 11 +- .../events/abstracts/models/persons.py | 7 +- .../events/abstracts/models/related_tracks.py | 4 +- .../abstracts/models/review_questions.py | 4 +- .../events/abstracts/models/review_ratings.py | 4 +- .../events/abstracts/models/reviews.py | 13 +- .../modules/events/abstracts/notifications.py | 11 +- .../events/abstracts/notifications_test.py | 6 +- indico/modules/events/abstracts/operations.py | 49 +- .../modules/events/abstracts/placeholders.py | 8 +- indico/modules/events/abstracts/schemas.py | 20 +- indico/modules/events/abstracts/settings.py | 8 +- .../templates/forms/track_role_widget.html | 2 +- .../management/abstract_list_filter.html | 6 +- .../management/cfa_actions/ended.html | 2 +- .../templates/reviewing/judgment.html | 2 +- .../abstracts/templates/reviewing/public.html | 6 +- indico/modules/events/abstracts/util.py | 85 +- indico/modules/events/abstracts/views.py | 4 +- indico/modules/events/agreements/__init__.py | 6 +- indico/modules/events/agreements/api.py | 8 +- indico/modules/events/agreements/base.py | 60 +- indico/modules/events/agreements/blueprint.py | 6 +- .../modules/events/agreements/controllers.py | 26 +- indico/modules/events/agreements/forms.py | 8 +- .../events/agreements/models/agreements.py | 12 +- .../agreements/models/agreements_test.py | 17 +- .../events/agreements/notifications.py | 4 +- .../modules/events/agreements/placeholders.py | 4 +- .../events/agreements/testing/fixtures.py | 8 +- indico/modules/events/agreements/util.py | 8 +- indico/modules/events/agreements/views.py | 4 +- indico/modules/events/api.py | 166 +- indico/modules/events/blueprint.py | 24 +- indico/modules/events/client/js/cloning.js | 2 +- indico/modules/events/client/js/creation.js | 2 +- indico/modules/events/client/js/display.js | 2 +- indico/modules/events/client/js/header.jsx | 50 + indico/modules/events/client/js/importing.js | 2 +- indico/modules/events/client/js/layout.js | 2 +- .../reviewing/components/TimelineContent.jsx | 6 +- .../js/reviewing/components/UserAvatar.jsx | 21 +- indico/modules/events/client/js/reviews.js | 2 +- indico/modules/events/client/js/roles.js | 52 +- .../events/client/js/util/list_generator.js | 2 +- .../modules/events/client/js/util/social.js | 2 +- .../events/client/js/util/static_filters.js | 2 +- .../events/client/js/util/types_dialog.js | 2 +- indico/modules/events/clone.py | 4 +- indico/modules/events/cloning.py | 50 +- indico/modules/events/cloning_test.py | 2 +- .../modules/events/contributions/__init__.py | 4 +- .../modules/events/contributions/blueprint.py | 10 +- .../client/js/PublicationSwitch.jsx | 9 +- .../events/contributions/client/js/index.jsx | 37 +- indico/modules/events/contributions/clone.py | 4 +- .../events/contributions/contrib_fields.py | 8 +- .../contributions/controllers/common.py | 6 +- .../contributions/controllers/compat.py | 14 +- .../contributions/controllers/display.py | 67 +- .../contributions/controllers/display_test.py | 52 + .../contributions/controllers/management.py | 127 +- indico/modules/events/contributions/fields.py | 14 +- indico/modules/events/contributions/forms.py | 24 +- indico/modules/events/contributions/ical.py | 46 + indico/modules/events/contributions/lists.py | 43 +- .../contributions/models/contributions.py | 31 +- .../models/contributions_test.py | 10 +- .../events/contributions/models/fields.py | 16 +- .../contributions/models/legacy_mapping.py | 12 +- .../events/contributions/models/persons.py | 10 +- .../events/contributions/models/principals.py | 7 +- .../events/contributions/models/references.py | 8 +- .../contributions/models/subcontributions.py | 23 +- .../events/contributions/models/types.py | 7 +- .../events/contributions/operations.py | 22 +- .../modules/events/contributions/schemas.py | 8 +- .../templates/contrib_list_filter.html | 4 +- .../templates/display/_contribution_list.html | 5 + .../display/contribution_display.html | 13 +- .../display/contribution_ical_export.html | 15 - .../templates/forms/contribution.html | 2 +- indico/modules/events/contributions/util.py | 71 +- .../modules/events/contributions/util_test.py | 4 +- indico/modules/events/contributions/views.py | 4 +- indico/modules/events/controllers/admin.py | 26 +- indico/modules/events/controllers/base.py | 6 +- .../modules/events/controllers/base_test.py | 16 +- indico/modules/events/controllers/creation.py | 12 +- indico/modules/events/controllers/display.py | 47 +- indico/modules/events/controllers/entry.py | 38 +- indico/modules/events/editing/__init__.py | 6 +- indico/modules/events/editing/blueprint.py | 16 +- .../js/editing/EditableSubmissionButton.jsx | 26 +- .../client/js/editing/ReduxTimeline.jsx | 19 +- .../editing/client/js/editing/index.jsx | 14 +- .../js/editing/page_layout/EditingView.jsx | 16 +- .../page_layout/EditingView.module.scss | 2 +- .../client/js/editing/page_layout/MenuBar.jsx | 77 +- .../editing/page_layout/MenuBar.module.scss | 2 +- .../client/js/editing/page_layout/index.js | 2 +- .../editing/timeline/ChangesConfirmation.jsx | 6 +- .../timeline/ChangesConfirmation.module.scss | 2 +- .../js/editing/timeline/CommentForm.jsx | 8 +- .../editing/timeline/CommentForm.module.scss | 2 +- .../js/editing/timeline/CommentItem.jsx | 30 +- .../js/editing/timeline/CustomActions.jsx | 99 + .../timeline/CustomActions.module.scss} | 8 +- .../client/js/editing/timeline/CustomItem.jsx | 19 +- .../FileDisplay/FileDisplay.module.scss | 2 +- .../js/editing/timeline/FileDisplay/index.jsx | 6 +- .../editing/timeline/FileManager/FileList.jsx | 8 +- .../FileManager/FileManager.module.scss | 2 +- .../editing/timeline/FileManager/Uploads.jsx | 4 +- .../__tests__/FileManager.spec.jsx | 7 +- .../editing/timeline/FileManager/actions.js | 2 +- .../js/editing/timeline/FileManager/index.jsx | 65 +- .../editing/timeline/FileManager/reducer.js | 3 +- .../editing/timeline/FileManager/selectors.js | 3 +- .../js/editing/timeline/FileManager/util.js | 24 +- .../js/editing/timeline/ResetReview.jsx | 9 +- .../editing/timeline/ResetReviews.module.scss | 2 +- .../client/js/editing/timeline/ReviewForm.jsx | 15 +- .../editing/timeline/ReviewForm.module.scss | 2 +- .../js/editing/timeline/RevisionLog.jsx | 19 +- .../js/editing/timeline/StateIndicator.jsx | 6 +- .../timeline/StateIndicator.module.scss | 2 +- .../js/editing/timeline/SubmitRevision.jsx | 23 +- .../js/editing/timeline/TimelineHeader.jsx | 22 +- .../js/editing/timeline/TimelineItem.jsx | 42 +- .../editing/timeline/TimelineItem.module.scss | 14 +- .../timeline/__tests__/selectors.test.js | 80 +- .../client/js/editing/timeline/actions.js | 2 +- .../client/js/editing/timeline/common.scss | 2 +- .../client/js/editing/timeline/index.jsx | 6 +- .../timeline/judgment/AcceptRejectForm.jsx | 12 +- .../editing/timeline/judgment/JudgmentBox.jsx | 9 +- .../timeline/judgment/JudgmentBox.module.scss | 2 +- .../judgment/JudgmentDropdownItems.jsx | 7 +- .../JudgmentDropdownItems.module.scss | 2 +- .../timeline/judgment/RequestChangesForm.jsx | 12 +- .../js/editing/timeline/judgment/TagInput.jsx | 5 +- .../timeline/judgment/TagInput.module.scss | 2 +- .../timeline/judgment/UpdateFilesForm.jsx | 18 +- .../client/js/editing/timeline/reducer.js | 4 +- .../client/js/editing/timeline/selectors.js | 131 +- .../client/js/editing/timeline/util.js | 90 +- .../modules/events/editing/client/js/index.js | 2 +- .../client/js/management/EditableTypeList.jsx | 22 +- .../management/EditableTypeList.module.scss | 2 +- .../js/management/EditingManagement.jsx | 21 +- .../management/EditingManagementDashboard.jsx | 12 +- .../client/js/management/ManageService.jsx | 51 +- .../editing/client/js/management/Section.jsx | 4 +- .../management/editable_type/EditableList.jsx | 80 +- .../editable_type/EditableList.module.scss | 2 +- .../editable_type/EditableTypeDashboard.jsx | 53 +- .../EditableTypeDashboard.module.scss | 2 +- .../editable_type/EditableTypeSubPageNav.jsx | 11 +- .../management/editable_type/NextEditable.jsx | 29 +- .../editable_type/NextEditable.module.scss | 2 +- .../management/editable_type/TeamManager.jsx | 15 +- .../file_types/ExtensionList.jsx | 5 +- .../file_types/FileTypeManager.jsx | 21 +- .../file_types/FileTypeManager.module.scss | 2 +- .../file_types/FileTypeModal.jsx | 6 +- .../editable_type/file_types/index.jsx | 9 +- .../js/management/editable_type/index.js | 2 +- .../review_conditions/ConditionInfo.jsx | 11 +- .../ConditionInfo.module.scss | 2 +- .../review_conditions/ReviewConditionForm.jsx | 5 +- .../ReviewConditionForm.module.scss | 2 +- .../ReviewConditionsManager.jsx | 11 +- .../ReviewConditionsManager.module.scss | 2 +- .../review_conditions/context.js | 2 +- .../editable_type/review_conditions/index.jsx | 15 +- .../editing/client/js/management/index.jsx | 2 +- .../client/js/management/tags/TagManager.jsx | 17 +- .../js/management/tags/TagManager.module.scss | 2 +- .../client/js/management/tags/TagModal.jsx | 4 +- .../client/js/management/tags/index.js | 8 +- .../events/editing/client/js/models.js | 2 +- indico/modules/events/editing/clone.py | 4 +- .../editing/controllers/backend/common.py | 17 +- .../controllers/backend/editable_list.py | 42 +- .../editing/controllers/backend/management.py | 25 +- .../editing/controllers/backend/service.py | 12 +- .../editing/controllers/backend/timeline.py | 127 +- .../events/editing/controllers/base.py | 17 +- .../events/editing/controllers/frontend.py | 6 +- indico/modules/events/editing/fields.py | 18 +- .../modules/events/editing/models/comments.py | 7 +- .../modules/events/editing/models/editable.py | 34 +- .../events/editing/models/file_types.py | 7 +- .../editing/models/review_conditions.py | 7 +- .../events/editing/models/revision_files.py | 9 +- .../events/editing/models/revisions.py | 9 +- indico/modules/events/editing/models/tags.py | 9 +- .../modules/events/editing/notifications.py | 8 +- indico/modules/events/editing/operations.py | 65 +- .../modules/events/editing/operations_test.py | 4 +- indico/modules/events/editing/schemas.py | 155 +- indico/modules/events/editing/service.py | 198 +- indico/modules/events/editing/settings.py | 5 +- .../templates/emails/comment_notification.txt | 2 +- .../emails/editor_judgment_notification.txt | 2 +- indico/modules/events/editing/util.py | 2 +- indico/modules/events/editing/views.py | 4 +- indico/modules/events/export.py | 110 +- indico/modules/events/export_test.py | 26 +- indico/modules/events/export_test_1.yaml | 250 +- indico/modules/events/export_test_2.yaml | 366 +- indico/modules/events/features/__init__.py | 23 +- indico/modules/events/features/base.py | 18 +- indico/modules/events/features/blueprint.py | 6 +- indico/modules/events/features/controllers.py | 16 +- .../events/features/templates/features.html | 2 +- indico/modules/events/features/util.py | 30 +- indico/modules/events/features/views.py | 4 +- indico/modules/events/fields.py | 44 +- indico/modules/events/forms.py | 14 +- indico/modules/events/ical.py | 130 + indico/modules/events/layout/__init__.py | 8 +- indico/modules/events/layout/blueprint.py | 14 +- indico/modules/events/layout/clone.py | 4 +- indico/modules/events/layout/compat.py | 8 +- .../events/layout/controllers/images.py | 6 +- .../events/layout/controllers/layout.py | 24 +- .../modules/events/layout/controllers/menu.py | 10 +- indico/modules/events/layout/forms.py | 20 +- indico/modules/events/layout/models/images.py | 9 +- .../events/layout/models/legacy_mapping.py | 12 +- indico/modules/events/layout/models/menu.py | 24 +- .../events/layout/templates/image_list.html | 2 +- .../layout/templates/layout_conference.html | 2 + indico/modules/events/layout/util.py | 75 +- indico/modules/events/layout/views.py | 4 +- indico/modules/events/legacy_ids_test.py | 50 + indico/modules/events/logs/__init__.py | 6 +- indico/modules/events/logs/blueprint.py | 6 +- .../modules/events/logs/client/js/actions.js | 2 +- .../logs/client/js/components/EventLog.jsx | 5 +- .../logs/client/js/components/Filter.jsx | 10 +- .../client/js/components/LogEntryList.jsx | 16 +- .../client/js/components/LogEntryModal.jsx | 4 +- .../logs/client/js/components/SearchBox.jsx | 4 +- .../logs/client/js/components/Toolbar.jsx | 4 +- .../logs/client/js/containers/Filter.js | 5 +- .../logs/client/js/containers/LogEntryList.js | 5 +- .../client/js/containers/LogEntryModal.js | 4 +- .../logs/client/js/containers/SearchBox.js | 5 +- .../modules/events/logs/client/js/index.jsx | 4 +- .../modules/events/logs/client/js/reducers.js | 2 +- .../events/logs/client/style/logs.scss | 10 +- indico/modules/events/logs/controllers.py | 8 +- indico/modules/events/logs/models/entries.py | 12 +- indico/modules/events/logs/renderers.py | 12 +- .../events/logs/templates/entry_simple.html | 2 +- indico/modules/events/logs/util.py | 27 +- indico/modules/events/logs/views.py | 4 +- indico/modules/events/management/__init__.py | 4 +- indico/modules/events/management/blueprint.py | 14 +- .../events/management/client/js/badges.js | 2 +- .../events/management/client/js/index.js | 105 +- .../events/management/controllers/__init__.py | 4 +- .../events/management/controllers/actions.py | 8 +- .../events/management/controllers/base.py | 22 +- .../events/management/controllers/cloning.py | 57 +- .../events/management/controllers/posters.py | 14 +- .../management/controllers/program_codes.py | 16 +- .../management/controllers/protection.py | 16 +- .../events/management/controllers/settings.py | 6 +- indico/modules/events/management/forms.py | 60 +- .../events/management/program_codes.py | 37 +- indico/modules/events/management/settings.py | 4 +- .../templates/_event_person_list.html | 2 +- .../templates/assign_program_codes.html | 2 +- .../management/templates/clone_event.html | 2 +- indico/modules/events/management/util.py | 20 +- indico/modules/events/management/views.py | 4 +- indico/modules/events/models/__init__.py | 4 +- indico/modules/events/models/acls_test.py | 53 +- indico/modules/events/models/events.py | 90 +- indico/modules/events/models/events_test.py | 6 +- indico/modules/events/models/labels.py | 7 +- .../modules/events/models/legacy_mapping.py | 10 +- indico/modules/events/models/persons.py | 48 +- indico/modules/events/models/principals.py | 7 +- indico/modules/events/models/references.py | 10 +- indico/modules/events/models/reviews.py | 30 +- indico/modules/events/models/roles.py | 15 +- indico/modules/events/models/series.py | 7 +- indico/modules/events/models/settings.py | 13 +- .../events/models/static_list_links.py | 11 +- indico/modules/events/module.json | 1 + indico/modules/events/notes/__init__.py | 6 +- indico/modules/events/notes/api.py | 6 +- indico/modules/events/notes/blueprint.py | 8 +- indico/modules/events/notes/clone.py | 4 +- indico/modules/events/notes/controllers.py | 16 +- indico/modules/events/notes/forms.py | 4 +- indico/modules/events/notes/models/notes.py | 20 +- .../modules/events/notes/models/notes_test.py | 4 +- .../events/notes/templates/edit_note.html | 2 +- indico/modules/events/notes/util.py | 8 +- indico/modules/events/notifications.py | 11 +- indico/modules/events/operations.py | 50 +- indico/modules/events/papers/__init__.py | 3 +- indico/modules/events/papers/blueprint.py | 14 +- .../events/papers/client/js/actions.js | 29 +- .../client/js/components/CommentForm.jsx | 8 +- .../client/js/components/GroupReviewForm.jsx | 8 +- .../js/components/GroupReviewForm.module.scss | 2 +- .../papers/client/js/components/Paper.jsx | 33 +- .../client/js/components/PaperContent.jsx | 3 +- .../js/components/PaperDecisionForm.jsx | 4 +- .../components/PaperDecisionForm.module.scss | 2 +- .../client/js/components/PaperFiles.jsx | 4 +- .../client/js/components/PaperMetadata.jsx | 7 +- .../client/js/components/PaperReviewForm.jsx | 7 +- .../js/components/PaperReviewForm.module.scss | 2 +- .../client/js/components/RevisionComment.jsx | 9 +- .../client/js/components/RevisionJudgment.jsx | 8 +- .../client/js/components/RevisionReview.jsx | 9 +- .../client/js/components/RevisionTimeline.jsx | 6 +- .../client/js/components/SubmitRevision.jsx | 6 +- .../client/js/components/TimelineHeader.jsx | 11 +- .../client/js/components/TimelineItem.jsx | 12 +- .../modules/events/papers/client/js/index.js | 6 +- .../modules/events/papers/client/js/models.js | 2 +- .../events/papers/client/js/reducers.js | 4 +- .../events/papers/client/js/selectors.js | 10 +- .../modules/events/papers/client/js/setup.jsx | 6 +- .../modules/events/papers/controllers/api.py | 18 +- .../modules/events/papers/controllers/base.py | 14 +- .../events/papers/controllers/display.py | 10 +- .../events/papers/controllers/management.py | 32 +- .../events/papers/controllers/paper.py | 75 +- .../events/papers/controllers/templates.py | 16 +- indico/modules/events/papers/fields.py | 6 +- indico/modules/events/papers/forms.py | 26 +- indico/modules/events/papers/lists.py | 64 +- .../events/papers/models/call_for_papers.py | 17 +- .../modules/events/papers/models/comments.py | 7 +- .../events/papers/models/competences.py | 7 +- indico/modules/events/papers/models/files.py | 13 +- indico/modules/events/papers/models/papers.py | 10 +- .../events/papers/models/review_questions.py | 4 +- .../events/papers/models/review_ratings.py | 4 +- .../modules/events/papers/models/reviews.py | 20 +- .../modules/events/papers/models/revisions.py | 15 +- .../modules/events/papers/models/templates.py | 9 +- .../papers/models/user_contributions.py | 4 +- indico/modules/events/papers/notifications.py | 4 +- indico/modules/events/papers/operations.py | 63 +- indico/modules/events/papers/schemas.py | 41 +- indico/modules/events/papers/settings.py | 6 +- .../papers/templates/_contributions.html | 2 +- .../events/papers/templates/_paper_list.html | 6 + .../management/paper_person_list.html | 2 +- .../papers/templates/paper_list_filter.html | 4 +- indico/modules/events/papers/util.py | 8 +- indico/modules/events/papers/views.py | 4 +- indico/modules/events/payment/__init__.py | 4 +- indico/modules/events/payment/blueprint.py | 16 +- indico/modules/events/payment/controllers.py | 36 +- indico/modules/events/payment/forms.py | 6 +- .../events/payment/models/transactions.py | 21 +- .../payment/models/transactions_test.py | 5 +- .../modules/events/payment/notifications.py | 4 +- indico/modules/events/payment/plugins.py | 25 +- .../templates/event_payment_cancel.html | 3 +- .../templates/event_payment_confirm.html | 2 +- .../events/payment/testing/fixtures.py | 6 +- indico/modules/events/payment/util.py | 14 +- indico/modules/events/payment/util_test.py | 5 +- indico/modules/events/payment/views.py | 4 +- indico/modules/events/persons/__init__.py | 10 +- indico/modules/events/persons/blueprint.py | 9 +- indico/modules/events/persons/controllers.py | 129 +- indico/modules/events/persons/forms.py | 12 +- indico/modules/events/persons/operations.py | 6 +- indico/modules/events/persons/placeholders.py | 37 +- indico/modules/events/persons/schemas.py | 21 + .../templates/emails/_contributions.html | 18 + .../management/_person_list_row.html | 4 +- .../templates/management/person_list.html | 14 +- indico/modules/events/persons/util.py | 15 +- indico/modules/events/persons/util_test.py | 6 +- indico/modules/events/persons/views.py | 4 +- indico/modules/events/posters.py | 10 +- .../modules/events/registration/__init__.py | 35 +- indico/modules/events/registration/api.py | 32 +- indico/modules/events/registration/badges.py | 34 +- .../modules/events/registration/blueprint.py | 21 +- .../registration/client/js/form/field.js | 4 +- .../registration/client/js/form/form.js | 26 +- .../registration/client/js/form/section.js | 4 +- .../client/js/form/sectiontoolbar.js | 2 +- .../registration/client/js/form/table.js | 2 +- .../registration/client/js/form/templates.js | 2 +- .../js/form/tpls/registrationform.tpl.html | 2 +- .../events/registration/client/js/index.js | 2 +- .../registration/client/js/invitations.js | 2 +- .../registration/client/js/registration.js | 2 +- .../events/registration/client/js/reglists.js | 55 +- indico/modules/events/registration/clone.py | 8 +- .../modules/events/registration/clone_test.py | 2 +- .../registration/controllers/__init__.py | 6 +- .../events/registration/controllers/compat.py | 10 +- .../registration/controllers/display.py | 62 +- .../registration/controllers/display_test.py | 10 +- .../controllers/management/__init__.py | 18 +- .../controllers/management/fields.py | 24 +- .../controllers/management/invitations.py | 10 +- .../controllers/management/regforms.py | 51 +- .../controllers/management/reglists.py | 165 +- .../controllers/management/sections.py | 25 +- .../controllers/management/tickets.py | 27 +- .../events/registration/fields/__init__.py | 13 +- .../events/registration/fields/base.py | 32 +- .../events/registration/fields/choices.py | 76 +- .../registration/fields/choices_test.py | 8 +- .../events/registration/fields/simple.py | 23 +- indico/modules/events/registration/forms.py | 99 +- indico/modules/events/registration/lists.py | 60 +- indico/modules/events/registration/logging.py | 11 +- .../events/registration/models/form_fields.py | 19 +- .../events/registration/models/forms.py | 48 +- .../events/registration/models/invitations.py | 17 +- .../events/registration/models/items.py | 62 +- .../registration/models/legacy_mapping.py | 9 +- .../registration/models/registrations.py | 72 +- .../events/registration/notifications.py | 22 +- .../registration/placeholders/invitations.py | 4 +- .../placeholders/registrations.py | 15 +- indico/modules/events/registration/schemas.py | 4 +- .../modules/events/registration/settings.py | 16 +- indico/modules/events/registration/stats.py | 76 +- .../templates/display/event_header.html | 4 +- .../templates/display/regform_display.html | 2 +- .../display/registration_modify.html | 6 +- .../display/registration_summary.html | 2 + .../emails/base_registration_details.html | 4 +- .../registration_creation_to_managers.html | 7 + .../registration_creation_to_registrant.html | 23 +- ...registration_state_update_to_managers.html | 1 + ...gistration_state_update_to_registrant.html | 1 + .../management/_registration_details.html | 8 +- .../templates/management/_reglist.html | 2 +- .../templates/management/regform_display.html | 2 +- .../templates/management/regform_modify.html | 2 +- .../templates/management/regform_reglist.html | 13 +- .../templates/management/regform_stats.html | 4 +- .../management/registration_details.html | 2 +- .../management/registration_modify.html | 2 +- .../templates/management/reglist_filter.html | 4 +- .../events/registration/testing/fixtures.py | 6 +- indico/modules/events/registration/util.py | 129 +- .../modules/events/registration/util_test.py | 37 +- indico/modules/events/registration/views.py | 4 +- indico/modules/events/reminders/__init__.py | 6 +- indico/modules/events/reminders/blueprint.py | 6 +- .../modules/events/reminders/controllers.py | 27 +- indico/modules/events/reminders/forms.py | 13 +- .../events/reminders/models/reminders.py | 31 +- indico/modules/events/reminders/tasks.py | 13 +- .../reminders/templates/edit_reminder.html | 3 +- indico/modules/events/reminders/util.py | 6 +- indico/modules/events/reminders/views.py | 4 +- indico/modules/events/requests/__init__.py | 8 +- indico/modules/events/requests/base.py | 8 +- indico/modules/events/requests/blueprint.py | 6 +- indico/modules/events/requests/controllers.py | 20 +- indico/modules/events/requests/exceptions.py | 5 +- .../events/requests/models/requests.py | 18 +- .../modules/events/requests/notifications.py | 23 +- indico/modules/events/requests/util.py | 10 +- indico/modules/events/requests/views.py | 4 +- .../events/reviewing_questions_fields.py | 8 +- indico/modules/events/roles/__init__.py | 4 +- indico/modules/events/roles/blueprint.py | 6 +- indico/modules/events/roles/clone.py | 4 +- indico/modules/events/roles/controllers.py | 53 +- indico/modules/events/roles/forms.py | 6 +- indico/modules/events/roles/util.py | 11 +- indico/modules/events/roles/views.py | 4 +- indico/modules/events/schemas.py | 6 +- indico/modules/events/sessions/__init__.py | 4 +- indico/modules/events/sessions/blueprint.py | 11 +- .../events/sessions/client/js/index.js | 4 +- .../sessions/client/js/session_display.jsx | 42 + indico/modules/events/sessions/clone.py | 4 +- .../events/sessions/controllers/compat.py | 8 +- .../events/sessions/controllers/display.py | 20 +- .../controllers/management/__init__.py | 10 +- .../controllers/management/sessions.py | 50 +- indico/modules/events/sessions/fields.py | 4 +- indico/modules/events/sessions/forms.py | 16 +- indico/modules/events/sessions/ical.py | 50 + .../modules/events/sessions/models/blocks.py | 14 +- .../events/sessions/models/legacy_mapping.py | 12 +- .../modules/events/sessions/models/persons.py | 9 +- .../events/sessions/models/principals.py | 7 +- .../events/sessions/models/sessions.py | 13 +- .../modules/events/sessions/models/types.py | 7 +- indico/modules/events/sessions/operations.py | 28 +- indico/modules/events/sessions/schemas.py | 28 + .../templates/display/session_display.html | 10 +- .../display/session_ical_export.html | 15 - indico/modules/events/sessions/util.py | 32 +- indico/modules/events/sessions/views.py | 4 +- indico/modules/events/settings.py | 44 +- indico/modules/events/static/__init__.py | 6 +- indico/modules/events/static/blueprint.py | 6 +- indico/modules/events/static/controllers.py | 4 +- indico/modules/events/static/models/static.py | 15 +- indico/modules/events/static/offline.py | 62 +- indico/modules/events/static/tasks.py | 14 +- indico/modules/events/static/util.py | 40 +- indico/modules/events/static/views.py | 4 +- indico/modules/events/surveys/__init__.py | 6 +- indico/modules/events/surveys/blueprint.py | 6 +- .../modules/events/surveys/client/js/index.js | 2 +- .../events/surveys/controllers/display.py | 8 +- .../controllers/management/__init__.py | 8 +- .../controllers/management/questionnaire.py | 97 +- .../surveys/controllers/management/results.py | 24 +- .../surveys/controllers/management/survey.py | 22 +- .../modules/events/surveys/fields/__init__.py | 10 +- indico/modules/events/surveys/fields/base.py | 4 +- .../modules/events/surveys/fields/choices.py | 24 +- .../modules/events/surveys/fields/simple.py | 14 +- indico/modules/events/surveys/forms.py | 16 +- indico/modules/events/surveys/models/items.py | 29 +- .../events/surveys/models/submissions.py | 11 +- .../modules/events/surveys/models/surveys.py | 20 +- indico/modules/events/surveys/operations.py | 4 +- indico/modules/events/surveys/placeholders.py | 4 +- indico/modules/events/surveys/tasks.py | 10 +- .../templates/management/survey_results.html | 4 +- indico/modules/events/surveys/util.py | 30 +- indico/modules/events/surveys/views.py | 4 +- .../templates/display/common/_legacy.html | 6 +- .../templates/display/conference/base.html | 13 +- .../templates/display/event_ical_export.html | 38 - .../templates/display/indico/_common.html | 6 +- indico/modules/events/templates/footer.html | 2 +- indico/modules/events/templates/header.html | 25 +- .../events/templates/management/_lists.html | 2 +- .../reviewing_questions_management.html | 2 +- .../events/templates/reviews/_common.html | 4 +- indico/modules/events/timetable/__init__.py | 6 +- indico/modules/events/timetable/blueprint.py | 6 +- indico/modules/events/timetable/clone.py | 4 +- .../events/timetable/controllers/__init__.py | 6 +- .../events/timetable/controllers/display.py | 5 +- .../timetable/controllers/display_test.py | 33 + .../events/timetable/controllers/legacy.py | 12 +- .../events/timetable/controllers/manage.py | 24 +- indico/modules/events/timetable/forms.py | 18 +- indico/modules/events/timetable/legacy.py | 38 +- .../modules/events/timetable/models/breaks.py | 7 +- .../events/timetable/models/entries.py | 29 +- indico/modules/events/timetable/operations.py | 16 +- indico/modules/events/timetable/reschedule.py | 16 +- .../templates/balloons/contribution.html | 2 +- .../timetable/templates/display/_weeks.html | 15 +- .../display/indico/_contribution.html | 5 +- .../display/indico/_subcontribution.html | 10 +- .../timetable/templates/move_entry.html | 4 +- .../events/timetable/testing/fixtures.py | 6 +- indico/modules/events/timetable/util.py | 41 +- indico/modules/events/timetable/util_test.py | 2 +- .../events/timetable/views/__init__.py | 6 +- .../modules/events/timetable/views/weeks.py | 10 +- indico/modules/events/tracks/__init__.py | 4 +- indico/modules/events/tracks/blueprint.py | 8 +- .../modules/events/tracks/client/js/index.js | 2 +- indico/modules/events/tracks/clone.py | 4 +- indico/modules/events/tracks/controllers.py | 12 +- indico/modules/events/tracks/forms.py | 6 +- indico/modules/events/tracks/models/groups.py | 7 +- .../events/tracks/models/principals.py | 7 +- indico/modules/events/tracks/models/tracks.py | 27 +- indico/modules/events/tracks/operations.py | 12 +- indico/modules/events/tracks/schemas.py | 6 +- indico/modules/events/tracks/settings.py | 4 +- indico/modules/events/tracks/views.py | 4 +- indico/modules/events/util.py | 160 +- indico/modules/events/views.py | 66 +- indico/modules/files/__init__.py | 4 +- indico/modules/files/blueprint.py | 7 +- indico/modules/files/controllers.py | 24 +- indico/modules/files/models/files.py | 20 +- indico/modules/files/schemas.py | 6 +- indico/modules/files/tasks.py | 4 +- indico/modules/groups/__init__.py | 4 +- indico/modules/groups/blueprint.py | 4 +- indico/modules/groups/controllers.py | 24 +- indico/modules/groups/core.py | 80 +- indico/modules/groups/forms.py | 8 +- indico/modules/groups/legacy.py | 69 - indico/modules/groups/models/groups.py | 10 +- indico/modules/groups/util.py | 7 +- indico/modules/groups/views.py | 4 +- indico/modules/legal/__init__.py | 4 +- indico/modules/legal/blueprint.py | 4 +- indico/modules/legal/controllers.py | 4 +- indico/modules/legal/forms.py | 4 +- indico/modules/legal/views.py | 4 +- indico/modules/networks/__init__.py | 6 +- indico/modules/networks/blueprint.py | 4 +- indico/modules/networks/controllers.py | 16 +- indico/modules/networks/fields.py | 15 +- indico/modules/networks/forms.py | 12 +- indico/modules/networks/models/networks.py | 16 +- indico/modules/networks/util.py | 9 +- indico/modules/networks/views.py | 4 +- indico/modules/news/__init__.py | 4 +- indico/modules/news/blueprint.py | 4 +- indico/modules/news/controllers.py | 4 +- indico/modules/news/forms.py | 4 +- indico/modules/news/models/news.py | 9 +- indico/modules/news/util.py | 6 +- indico/modules/news/views.py | 4 +- indico/modules/oauth/__init__.py | 34 +- indico/modules/oauth/blueprint.py | 15 +- indico/modules/oauth/controllers.py | 175 +- indico/modules/oauth/forms.py | 22 +- indico/modules/oauth/models/__init__.py | 0 indico/modules/oauth/models/applications.py | 170 - indico/modules/oauth/models/tokens.py | 144 - indico/modules/oauth/models/tokens_test.py | 100 - indico/modules/oauth/provider.py | 112 - indico/modules/oauth/provider_test.py | 158 - .../modules/oauth/templates/app_details.html | 33 +- .../modules/oauth/templates/user_profile.html | 17 +- indico/modules/oauth/testing/__init__.py | 0 indico/modules/oauth/testing/fixtures.py | 59 - indico/modules/oauth/views.py | 4 +- indico/modules/rb/__init__.py | 6 +- indico/modules/rb/api.py | 38 +- indico/modules/rb/blueprint.py | 16 +- indico/modules/rb/client/js/actions.js | 5 +- .../js/common/bookings/BookingDetails.jsx | 67 +- .../bookings/BookingDetails.module.scss | 2 +- .../common/bookings/BookingDetailsModal.jsx | 5 +- .../bookings/BookingDetailsPreloader.jsx | 7 +- .../client/js/common/bookings/BookingEdit.jsx | 20 +- .../common/bookings/BookingEdit.module.scss | 2 +- .../common/bookings/BookingEditCalendar.jsx | 6 +- .../bookings/BookingEditCalendar.module.scss | 2 +- .../js/common/bookings/BookingEditForm.jsx | 17 +- .../bookings/BookingEditForm.module.scss | 2 +- .../js/common/bookings/BookingExportModal.jsx | 10 +- .../js/common/bookings/BookingObjectLink.jsx | 6 +- .../bookings/BookingObjectLink.module.scss | 2 +- .../common/bookings/LazyBookingObjectLink.jsx | 6 +- .../js/common/bookings/OccurrencesCounter.jsx | 4 +- .../bookings/OccurrencesCounter.module.scss | 2 +- .../rb/client/js/common/bookings/actions.js | 5 +- .../rb/client/js/common/bookings/index.js | 2 +- .../rb/client/js/common/bookings/modals.jsx | 5 +- .../rb/client/js/common/bookings/reducers.js | 4 +- .../rb/client/js/common/bookings/selectors.js | 4 +- .../rb/client/js/common/config/actions.js | 3 +- .../rb/client/js/common/config/index.js | 2 +- .../rb/client/js/common/config/reducers.js | 6 +- .../rb/client/js/common/config/selectors.js | 2 +- .../rb/client/js/common/filters/FilterBar.jsx | 5 +- .../js/common/filters/FilterDropdown.jsx | 5 +- .../common/filters/FilterDropdown.module.scss | 2 +- .../js/common/filters/FilterFormComponent.jsx | 4 +- .../rb/client/js/common/filters/actions.js | 2 +- .../rb/client/js/common/filters/index.js | 2 +- .../rb/client/js/common/filters/reducers.js | 3 +- .../rb/client/js/common/filters/validation.js | 8 +- .../rb/client/js/common/linking/LinkBar.jsx | 10 +- .../js/common/linking/LinkBar.module.scss | 2 +- .../rb/client/js/common/linking/actions.js | 3 +- .../rb/client/js/common/linking/index.js | 2 +- .../rb/client/js/common/linking/props.js | 2 +- .../rb/client/js/common/linking/reducers.js | 5 +- .../rb/client/js/common/linking/selectors.js | 2 +- .../rb/client/js/common/map/MapController.jsx | 14 +- .../js/common/map/MapController.module.scss | 2 +- .../rb/client/js/common/map/MapMarkers.jsx | 7 +- .../client/js/common/map/RoomBookingMap.jsx | 6 +- .../js/common/map/RoomBookingMap.module.scss | 2 +- .../js/common/map/RoomBookingMapControl.jsx | 4 +- .../rb/client/js/common/map/actions.js | 5 +- .../modules/rb/client/js/common/map/index.js | 2 +- .../rb/client/js/common/map/reducers.js | 7 +- .../rb/client/js/common/map/selectors.js | 4 +- .../modules/rb/client/js/common/map/util.js | 9 +- .../rb/client/js/common/roomSearch/actions.js | 9 +- .../rb/client/js/common/roomSearch/index.js | 2 +- .../js/common/roomSearch/queryString.js | 9 +- .../client/js/common/roomSearch/reducers.js | 10 +- .../client/js/common/roomSearch/selectors.js | 4 +- .../js/common/roomSearch/serializers.js | 2 +- .../js/common/rooms/RoomDetailsModal.jsx | 29 +- .../common/rooms/RoomDetailsModal.module.scss | 2 +- .../js/common/rooms/RoomDetailsPreloader.jsx | 7 +- .../client/js/common/rooms/RoomRenderer.jsx | 8 +- .../js/common/rooms/RoomRenderer.module.scss | 2 +- .../rb/client/js/common/rooms/RoomStats.jsx | 8 +- .../js/common/rooms/RoomStats.module.scss | 2 +- .../rooms/__tests__/RoomEditModal.spec.jsx | 12 +- .../rb/client/js/common/rooms/actions.js | 7 +- .../common/rooms/edit/DailyAvailability.jsx | 12 +- .../rooms/edit/DailyAvailability.module.scss | 2 +- .../js/common/rooms/edit/EquipmentList.jsx | 8 +- .../rooms/edit/EquipmentList.module.scss | 2 +- .../common/rooms/edit/NonBookablePeriods.jsx | 13 +- .../js/common/rooms/edit/RoomEditDetails.jsx | 7 +- .../js/common/rooms/edit/RoomEditLocation.jsx | 7 +- .../js/common/rooms/edit/RoomEditModal.jsx | 29 +- .../rooms/edit/RoomEditModal.module.scss | 2 +- .../rooms/edit/RoomEditNotifications.jsx | 9 +- .../js/common/rooms/edit/RoomEditOptions.jsx | 14 +- .../common/rooms/edit/RoomEditPermissions.jsx | 9 +- .../client/js/common/rooms/edit/RoomPhoto.jsx | 10 +- .../common/rooms/edit/RoomPhoto.module.scss | 2 +- .../js/common/rooms/edit/TabPaneError.jsx | 4 +- .../rb/client/js/common/rooms/index.js | 4 +- .../rb/client/js/common/rooms/modals.jsx | 4 +- .../rb/client/js/common/rooms/reducers.js | 8 +- .../rb/client/js/common/rooms/selectors.js | 5 +- .../common/timeline/DailyTimelineContent.jsx | 12 +- .../js/common/timeline/DateNavigator.jsx | 6 +- .../common/timeline/DateNavigator.module.scss | 2 +- .../common/timeline/EditableTimelineItem.jsx | 5 +- .../js/common/timeline/ElasticTimeline.jsx | 8 +- .../timeline/MonthlyTimelineContent.jsx | 6 +- .../js/common/timeline/RowActionsDropdown.jsx | 11 +- .../timeline/RowActionsDropdown.module.scss | 2 +- .../timeline/SingleRoomTimelineModal.jsx | 10 +- .../SingleRoomTimelineModal.module.scss | 2 +- .../js/common/timeline/Timeline.module.scss | 2 +- .../timeline/TimelineContent.module.scss | 2 +- .../js/common/timeline/TimelineHeader.jsx | 9 +- .../js/common/timeline/TimelineItem.jsx | 8 +- .../common/timeline/TimelineItem.module.scss | 2 +- .../js/common/timeline/TimelineLegend.jsx | 3 +- .../timeline/TimelineLegend.module.scss | 2 +- .../common/timeline/WeeklyTimelineContent.jsx | 6 +- .../WeeklyTimelineContent.module.scss | 2 +- .../rb/client/js/common/timeline/index.js | 2 +- .../rb/client/js/common/timeline/reducers.js | 2 +- .../rb/client/js/common/user/actions.js | 4 +- .../modules/rb/client/js/common/user/index.js | 2 +- .../rb/client/js/common/user/reducers.js | 7 +- .../rb/client/js/common/user/selectors.js | 2 +- .../client/js/components/AdminOverrideBar.jsx | 8 +- .../components/AdminOverrideBar.module.scss | 2 +- .../modules/rb/client/js/components/App.jsx | 48 +- .../rb/client/js/components/App.module.scss | 2 +- .../js/components/BookingBootstrapForm.jsx | 11 +- .../BookingBootstrapForm.module.scss | 2 +- .../client/js/components/CardPlaceholder.jsx | 4 +- .../rb/client/js/components/DimmableImage.jsx | 4 +- .../js/components/DimmableImage.module.scss | 2 +- .../client/js/components/ItemPlaceholder.jsx | 4 +- .../modules/rb/client/js/components/Menu.jsx | 7 +- .../rb/client/js/components/Menu.module.scss | 2 +- .../rb/client/js/components/MenuItem.jsx | 5 +- .../client/js/components/MenuItem.module.scss | 2 +- .../modules/rb/client/js/components/Room.jsx | 14 +- .../rb/client/js/components/Room.module.scss | 2 +- .../client/js/components/RoomBasicDetails.jsx | 6 +- .../components/RoomBasicDetails.module.scss | 2 +- .../client/js/components/RoomFeatureEntry.jsx | 4 +- .../client/js/components/RoomKeyLocation.jsx | 5 +- .../js/components/RoomKeyLocation.module.scss | 2 +- .../rb/client/js/components/RoomSelector.jsx | 11 +- .../js/components/RoomSelector.module.scss | 2 +- .../rb/client/js/components/SearchBar.jsx | 9 +- .../js/components/SearchBar.module.scss | 2 +- .../rb/client/js/components/SidebarFooter.jsx | 12 +- .../rb/client/js/components/SidebarMenu.jsx | 16 +- .../js/components/SidebarMenu.module.scss | 2 +- .../rb/client/js/components/SpriteImage.jsx | 4 +- .../client/js/components/TimeInformation.jsx | 6 +- .../js/components/TimeInformation.module.scss | 2 +- .../client/js/components/TimeRangePicker.jsx | 7 +- .../js/components/TimeRangePicker.module.scss | 2 +- indico/modules/rb/client/js/history.js | 2 +- indico/modules/rb/client/js/index.jsx | 2 +- .../rb/client/js/modals/ModalController.jsx | 12 +- indico/modules/rb/client/js/modals/index.js | 2 +- .../rb/client/js/modules/admin/AdminArea.jsx | 19 +- .../js/modules/admin/AdminArea.module.scss | 2 +- .../js/modules/admin/AdminLocationRooms.jsx | 13 +- .../admin/AdminLocationRooms.module.scss | 2 +- .../rb/client/js/modules/admin/AdminMenu.jsx | 13 +- .../js/modules/admin/AdminMenu.module.scss | 2 +- .../client/js/modules/admin/AdminRoomItem.jsx | 9 +- .../modules/admin/AdminRoomItem.module.scss | 2 +- .../js/modules/admin/AttributesPage.jsx | 12 +- .../client/js/modules/admin/CategoryList.jsx | 5 +- .../client/js/modules/admin/EditableList.jsx | 6 +- .../js/modules/admin/EditableList.module.scss | 2 +- .../js/modules/admin/EditableListItem.jsx | 7 +- .../client/js/modules/admin/EquipmentPage.jsx | 9 +- .../js/modules/admin/EquipmentTypeList.jsx | 10 +- .../client/js/modules/admin/LocationPage.jsx | 16 +- .../client/js/modules/admin/MapAreasPage.jsx | 12 +- .../js/modules/admin/MapAreasPage.module.scss | 2 +- .../js/modules/admin/RoomFeatureList.jsx | 10 +- .../client/js/modules/admin/SettingsPage.jsx | 14 +- .../js/modules/admin/SettingsPage.module.scss | 2 +- .../rb/client/js/modules/admin/actions.js | 13 +- .../rb/client/js/modules/admin/index.js | 2 +- .../rb/client/js/modules/admin/reducers.js | 7 +- .../rb/client/js/modules/admin/selectors.js | 3 +- .../js/modules/blockings/BlockingCard.jsx | 6 +- .../blockings/BlockingCard.module.scss | 2 +- .../modules/blockings/BlockingFilterBar.jsx | 11 +- .../js/modules/blockings/BlockingList.jsx | 13 +- .../blockings/BlockingList.module.scss | 2 +- .../js/modules/blockings/BlockingModal.jsx | 17 +- .../blockings/BlockingModal.module.scss | 2 +- .../modules/blockings/BlockingPreloader.jsx | 7 +- .../rb/client/js/modules/blockings/actions.js | 14 +- .../rb/client/js/modules/blockings/index.js | 2 +- .../rb/client/js/modules/blockings/modals.jsx | 4 +- .../js/modules/blockings/queryString.js | 9 +- .../client/js/modules/blockings/reducers.js | 6 +- .../client/js/modules/blockings/selectors.js | 4 +- .../js/modules/blockings/serializers.js | 2 +- .../js/modules/bookRoom/BookFromListModal.jsx | 13 +- .../client/js/modules/bookRoom/BookRoom.jsx | 42 +- .../js/modules/bookRoom/BookRoom.module.scss | 2 +- .../js/modules/bookRoom/BookRoomModal.jsx | 31 +- .../bookRoom/BookRoomModal.module.scss | 2 +- .../js/modules/bookRoom/BookingFilterBar.jsx | 22 +- .../js/modules/bookRoom/BookingSuggestion.jsx | 5 +- .../bookRoom/BookingSuggestion.module.scss | 2 +- .../js/modules/bookRoom/BookingTimeline.jsx | 9 +- .../js/modules/bookRoom/SearchResultCount.jsx | 5 +- .../bookRoom/UnavailableRoomsModal.jsx | 19 +- .../rb/client/js/modules/bookRoom/actions.js | 17 +- .../js/modules/bookRoom/filters/DateForm.jsx | 6 +- .../modules/bookRoom/filters/DateRenderer.jsx | 2 +- .../bookRoom/filters/RecurrenceForm.jsx | 4 +- .../filters/RecurrenceForm.module.scss | 2 +- .../js/modules/bookRoom/filters/TimeForm.jsx | 5 +- .../bookRoom/filters/TimeForm.module.scss | 2 +- .../modules/bookRoom/filters/TimeRenderer.jsx | 2 +- .../rb/client/js/modules/bookRoom/index.js | 2 +- .../rb/client/js/modules/bookRoom/modals.jsx | 12 +- .../client/js/modules/bookRoom/queryString.js | 11 +- .../rb/client/js/modules/bookRoom/reducers.js | 6 +- .../client/js/modules/bookRoom/selectors.js | 6 +- .../client/js/modules/bookRoom/serializers.js | 2 +- .../client/js/modules/calendar/Calendar.jsx | 19 +- .../js/modules/calendar/Calendar.module.scss | 2 +- .../js/modules/calendar/CalendarListView.jsx | 12 +- .../calendar/CalendarListView.module.scss | 2 +- .../rb/client/js/modules/calendar/actions.js | 15 +- .../rb/client/js/modules/calendar/index.js | 2 +- .../client/js/modules/calendar/queryString.js | 9 +- .../rb/client/js/modules/calendar/reducers.js | 8 +- .../client/js/modules/calendar/selectors.js | 4 +- .../client/js/modules/calendar/serializers.js | 2 +- .../rb/client/js/modules/landing/Landing.jsx | 16 +- .../js/modules/landing/Landing.module.scss | 2 +- .../js/modules/landing/LandingStatistics.jsx | 9 +- .../js/modules/landing/UpcomingBookings.jsx | 10 +- .../landing/UpcomingBookings.module.scss | 2 +- .../rb/client/js/modules/landing/actions.js | 5 +- .../rb/client/js/modules/landing/index.js | 2 +- .../rb/client/js/modules/landing/reducers.js | 6 +- .../rb/client/js/modules/landing/selectors.js | 3 +- .../js/modules/roomList/RoomFilterBar.jsx | 19 +- .../roomList/RoomFilterBar.module.scss | 2 +- .../client/js/modules/roomList/RoomList.jsx | 25 +- .../js/modules/roomList/RoomList.module.scss | 2 +- .../rb/client/js/modules/roomList/actions.js | 2 +- .../modules/roomList/filters/BuildingForm.jsx | 5 +- .../modules/roomList/filters/CapacityForm.jsx | 3 +- .../roomList/filters/CapacityForm.module.scss | 2 +- .../roomList/filters/EquipmentForm.jsx | 6 +- .../filters/EquipmentForm.module.scss | 2 +- .../modules/roomList/filters/ShowOnlyForm.jsx | 4 +- .../roomList/filters/ShowOnlyForm.module.scss | 2 +- .../rb/client/js/modules/roomList/index.js | 2 +- .../client/js/modules/roomList/queryString.js | 2 +- .../rb/client/js/modules/roomList/reducers.js | 2 +- .../client/js/modules/roomList/selectors.js | 2 +- indico/modules/rb/client/js/props.js | 2 +- indico/modules/rb/client/js/reducers.js | 10 +- indico/modules/rb/client/js/selectors.js | 2 +- indico/modules/rb/client/js/serializers.js | 2 +- indico/modules/rb/client/js/setup.jsx | 16 +- indico/modules/rb/client/js/store.js | 19 +- indico/modules/rb/client/js/util.jsx | 11 +- indico/modules/rb/client/styles/main.scss | 2 +- indico/modules/rb/client/styles/palette.scss | 2 +- .../modules/rb/client/styles/responsive.scss | 2 +- .../modules/rb/client/styles/sui_fixes.scss | 2 +- indico/modules/rb/client/styles/util.scss | 2 +- indico/modules/rb/controllers/__init__.py | 6 +- .../modules/rb/controllers/backend/admin.py | 8 +- .../rb/controllers/backend/blockings.py | 6 +- .../rb/controllers/backend/bookings.py | 52 +- .../modules/rb/controllers/backend/common.py | 8 +- .../rb/controllers/backend/locations.py | 4 +- indico/modules/rb/controllers/backend/misc.py | 15 +- .../modules/rb/controllers/backend/rooms.py | 8 +- indico/modules/rb/controllers/frontend.py | 4 +- indico/modules/rb/event/client/js/index.js | 2 +- indico/modules/rb/event/controllers.py | 7 +- indico/modules/rb/event/fields.py | 18 +- indico/modules/rb/event/forms.py | 6 +- indico/modules/rb/models/blocked_rooms.py | 21 +- .../modules/rb/models/blocked_rooms_test.py | 4 +- .../modules/rb/models/blocking_principals.py | 6 +- indico/modules/rb/models/blockings.py | 11 +- indico/modules/rb/models/blockings_test.py | 4 +- indico/modules/rb/models/equipment.py | 7 +- indico/modules/rb/models/favorites.py | 4 +- indico/modules/rb/models/locations.py | 7 +- indico/modules/rb/models/locations_test.py | 4 +- indico/modules/rb/models/map_areas.py | 7 +- indico/modules/rb/models/photos.py | 6 +- indico/modules/rb/models/principals.py | 7 +- .../rb/models/reservation_edit_logs.py | 6 +- .../rb/models/reservation_occurrences.py | 39 +- .../rb/models/reservation_occurrences_test.py | 20 +- indico/modules/rb/models/reservations.py | 94 +- indico/modules/rb/models/reservations_test.py | 2 +- indico/modules/rb/models/room_attributes.py | 8 +- .../modules/rb/models/room_bookable_hours.py | 6 +- .../rb/models/room_bookable_hours_test.py | 2 +- indico/modules/rb/models/room_features.py | 7 +- .../rb/models/room_nonbookable_periods.py | 6 +- .../models/room_nonbookable_periods_test.py | 2 +- indico/modules/rb/models/rooms.py | 46 +- indico/modules/rb/models/rooms_test.py | 125 +- indico/modules/rb/models/util.py | 4 +- indico/modules/rb/models/util_test.py | 2 +- indico/modules/rb/notifications/blockings.py | 14 +- .../notifications/reservation_occurrences.py | 16 +- .../modules/rb/notifications/reservations.py | 41 +- indico/modules/rb/operations/admin.py | 8 +- indico/modules/rb/operations/blockings.py | 10 +- indico/modules/rb/operations/bookings.py | 36 +- indico/modules/rb/operations/bookings_test.py | 2 +- indico/modules/rb/operations/conflicts.py | 24 +- indico/modules/rb/operations/misc.py | 13 +- indico/modules/rb/operations/rooms.py | 9 +- indico/modules/rb/operations/rooms_test.py | 26 +- indico/modules/rb/operations/suggestions.py | 9 +- indico/modules/rb/schemas.py | 58 +- indico/modules/rb/settings.py | 6 +- indico/modules/rb/statistics.py | 19 +- indico/modules/rb/tasks.py | 4 +- indico/modules/rb/tasks_test.py | 6 +- .../reminders/finishing_bookings.html | 2 +- indico/modules/rb/testing/fixtures.py | 38 +- indico/modules/rb/user_prefs.py | 4 +- indico/modules/rb/util.py | 62 +- indico/modules/rb/util_test.py | 8 +- indico/modules/rb/views.py | 4 +- indico/modules/users/__init__.py | 4 +- indico/modules/users/api.py | 46 +- indico/modules/users/blueprint.py | 41 +- indico/modules/users/client/js/Favorites.jsx | 207 + .../users/client/js/Favorites.module.scss | 47 + .../users/client/js/ICSCalendarLink.jsx | 142 - .../users/client/js/ProfilePicture.jsx | 16 +- .../client/js/ProfilePicture.module.scss | 2 +- indico/modules/users/client/js/dashboard.jsx | 13 +- indico/modules/users/client/js/index.js | 2 +- indico/modules/users/controllers.py | 220 +- indico/modules/users/ext.py | 23 +- indico/modules/users/forms.py | 24 +- indico/modules/users/legacy.py | 335 - indico/modules/users/models/affiliations.py | 8 +- indico/modules/users/models/emails.py | 8 +- indico/modules/users/models/favorites.py | 4 +- indico/modules/users/models/settings.py | 30 +- indico/modules/users/models/suggestions.py | 7 +- indico/modules/users/models/users.py | 103 +- indico/modules/users/models/users_test.py | 39 +- indico/modules/users/module.json | 3 +- indico/modules/users/operations.py | 4 +- indico/modules/users/schemas.py | 16 +- indico/modules/users/tasks.py | 4 +- indico/modules/users/templates/_category.html | 11 - .../modules/users/templates/_favorites.html | 12 - indico/modules/users/templates/dashboard.html | 17 +- .../users/templates/emails/verify_email.txt | 12 +- indico/modules/users/templates/favorites.html | 87 +- .../modules/users/templates/users_admin.html | 1 + indico/modules/users/util.py | 91 +- indico/modules/users/views.py | 8 +- indico/modules/vc/__init__.py | 6 +- indico/modules/vc/blueprint.py | 24 +- indico/modules/vc/client/js/index.js | 42 +- indico/modules/vc/clone.py | 18 +- indico/modules/vc/controllers.py | 65 +- indico/modules/vc/exceptions.py | 9 +- indico/modules/vc/forms.py | 42 +- indico/modules/vc/models/__init__.py | 4 +- indico/modules/vc/models/vc_rooms.py | 41 +- indico/modules/vc/notifications.py | 8 +- indico/modules/vc/plugins.py | 63 +- indico/modules/vc/templates/manage_event.html | 3 + .../vc/templates/manage_event_select.html | 2 +- indico/modules/vc/util.py | 17 +- indico/modules/vc/views.py | 4 +- indico/testing/fixtures/app.py | 34 +- indico/testing/fixtures/cache.py | 16 + indico/testing/fixtures/category.py | 10 +- indico/testing/fixtures/contribution.py | 12 +- indico/testing/fixtures/database.py | 34 +- indico/testing/fixtures/disallow.py | 6 +- indico/testing/fixtures/event.py | 8 +- indico/testing/fixtures/person.py | 4 +- indico/testing/fixtures/session.py | 31 + indico/testing/fixtures/smtp.py | 16 +- indico/testing/fixtures/storage.py | 6 +- indico/testing/fixtures/user.py | 23 +- indico/testing/fixtures/util.py | 8 +- indico/testing/pytest_plugin.py | 12 +- indico/testing/util.py | 22 +- .../fr_FR/LC_MESSAGES/messages-js.po | 30 +- .../fr_FR/LC_MESSAGES/messages-react.po | 1767 +- .../fr_FR/LC_MESSAGES/messages.po | 1215 +- indico/translations/messages-js.pot | 28 +- indico/translations/messages-react.pot | 1756 +- indico/translations/messages.pot | 1209 +- .../pl_PL/LC_MESSAGES/messages-js.po | 2040 + .../pl_PL/LC_MESSAGES/messages-react.po | 4029 ++ .../pl_PL/LC_MESSAGES/messages.po | 17346 ++++++ .../uk_UA/LC_MESSAGES/messages-js.po | 2036 + .../uk_UA/LC_MESSAGES/messages-react.po | 4030 ++ .../uk_UA/LC_MESSAGES/messages.po | 17309 ++++++ .../zh_Hans_CN/LC_MESSAGES/messages-js.po | 30 +- .../zh_Hans_CN/LC_MESSAGES/messages-react.po | 1766 +- .../zh_Hans_CN/LC_MESSAGES/messages.po | 971 +- indico/util/benchmark.py | 10 +- indico/util/caching.py | 18 +- indico/util/caching_test.py | 31 +- indico/util/console.py | 62 +- indico/util/countries.py | 4 +- indico/util/date_time.py | 147 +- indico/util/date_time_test.py | 2 +- indico/util/decorators.py | 7 +- indico/util/emails/__init__.py | 0 indico/util/emails/backend.py | 200 - indico/util/{struct => }/enum.py | 6 +- indico/util/event.py | 12 +- indico/util/event_test.py | 2 +- indico/util/fossilize/__init__.py | 412 - indico/util/fossilize/conversion.py | 46 - indico/util/fs.py | 14 +- indico/util/fs_test.py | 29 +- indico/util/i18n.py | 176 +- indico/util/i18n_test.py | 57 +- indico/util/images.py | 2 +- indico/util/{struct => }/iterables.py | 16 +- indico/util/iterables_test.py | 16 + indico/util/json.py | 20 +- indico/util/locators.py | 16 +- indico/util/locators_test.py | 12 +- indico/util/marshmallow.py | 105 +- indico/util/marshmallow_test.py | 10 +- indico/util/mathjax.py | 8 +- indico/util/mdx_latex.py | 72 +- indico/util/mdx_latex_test.py | 4 +- indico/util/mimetypes.py | 10 +- indico/util/mimetypes_test.py | 2 +- indico/util/network.py | 4 +- indico/util/network_test.py | 2 +- indico/util/packaging.py | 20 +- indico/util/passwords.py | 155 +- indico/util/passwords_test.py | 99 +- indico/util/placeholders.py | 34 +- indico/util/process.py | 2 +- indico/util/roles.py | 20 +- indico/util/rules.py | 16 +- indico/util/rules_test.py | 8 +- indico/util/serializer.py | 11 +- indico/util/signals.py | 14 +- indico/util/signals_test.py | 10 +- indico/util/signing.py | 2 +- indico/util/spreadsheets.py | 64 +- indico/util/spreadsheets_test.py | 6 +- indico/util/string.py | 284 +- indico/util/string_test.py | 47 +- indico/util/struct/__init__.py | 0 indico/util/struct/iterables_test.py | 16 - indico/util/suggestions.py | 25 +- indico/util/system.py | 15 +- indico/util/tasks.py | 5 +- indico/util/user.py | 151 +- indico/util/user_test.py | 4 +- .../implementation => vendor}/__init__.py | 0 indico/vendor/django_mail/__init__.py | 39 + .../django_mail/backends}/__init__.py | 0 indico/vendor/django_mail/backends/base.py | 77 + indico/vendor/django_mail/backends/console.py | 58 + indico/vendor/django_mail/backends/locmem.py | 46 + indico/vendor/django_mail/backends/smtp.py | 183 + indico/vendor/django_mail/encoding_utils.py | 79 + .../emails => vendor/django_mail}/message.py | 359 +- .../django_mail/module_loading_utils.py | 37 + indico/vendor/django_mail/utils.py | 36 + indico/web/args.py | 68 +- indico/web/args_test.py | 335 + indico/web/assets/blueprint.py | 47 +- indico/web/assets/util.py | 6 +- indico/web/assets/vars_js.py | 24 +- indico/web/breadcrumbs.py | 6 +- indico/web/client/js/exports.js | 2 +- indico/web/client/js/index.js | 2 +- indico/web/client/js/jquery/ckeditor.js | 2 +- indico/web/client/js/jquery/compat/jqplot.js | 2 +- .../web/client/js/jquery/compat/jquery-ui.js | 2 +- indico/web/client/js/jquery/compat/mathjax.js | 2 +- .../client/js/jquery/extensions/clipboard.js | 2 +- .../web/client/js/jquery/extensions/global.js | 2 +- indico/web/client/js/jquery/index.js | 4 +- indico/web/client/js/jquery/markdown.js | 2 +- indico/web/client/js/jquery/statistics.js | 2 +- .../web/client/js/jquery/utils/ajaxdialog.js | 2 +- indico/web/client/js/jquery/utils/ajaxform.js | 2 +- .../web/client/js/jquery/utils/declarative.js | 2 +- indico/web/client/js/jquery/utils/defaults.js | 2 +- indico/web/client/js/jquery/utils/dropzone.js | 2 +- indico/web/client/js/jquery/utils/errors.js | 2 +- indico/web/client/js/jquery/utils/forms.js | 2 +- indico/web/client/js/jquery/utils/misc.js | 2 +- .../js/jquery/utils/pagedown_mathjax.js | 2 +- indico/web/client/js/jquery/utils/routing.js | 2 +- .../client/js/jquery/utils/sortablelist.js | 2 +- .../client/js/jquery/widgets/actioninput.js | 2 +- .../client/js/jquery/widgets/ajaxcheckbox.js | 2 +- .../client/js/jquery/widgets/ajaxqbubble.js | 2 +- .../js/jquery/widgets/categorynavigator.js | 387 +- .../js/jquery/widgets/clearableinput.js | 2 +- .../client/js/jquery/widgets/colorpicker.js | 2 +- .../web/client/js/jquery/widgets/dropdown.js | 2 +- .../web/client/js/jquery/widgets/dttbutton.js | 2 +- indico/web/client/js/jquery/widgets/index.js | 2 +- .../client/js/jquery/widgets/itempicker.js | 2 +- .../widgets/jinja/category_picker_widget.js | 38 +- .../jquery/widgets/jinja/ckeditor_widget.js | 2 +- .../widgets/jinja/color_picker_widget.js | 2 +- .../js/jquery/widgets/jinja/date_widget.js | 2 +- .../jquery/widgets/jinja/datetime_widget.js | 3 +- .../client/js/jquery/widgets/jinja/index.js | 2 +- .../js/jquery/widgets/jinja/linking_widget.js | 2 +- .../jquery/widgets/jinja/location_widget.js | 2 +- .../jquery/widgets/jinja/markdown_widget.js | 2 +- .../widgets/jinja/multiple_items_widget.js | 2 +- .../widgets/jinja/occurrences_widget.js | 2 +- .../jinja/override_multiple_items_widget.js | 2 +- .../widgets/jinja/permissions_widget.js | 101 +- .../widgets/jinja/person_link_widget.js | 200 +- .../widgets/jinja/principal_list_widget.js | 2 +- .../jquery/widgets/jinja/principal_widget.js | 4 +- .../jquery/widgets/jinja/protection_widget.js | 25 +- .../jquery/widgets/jinja/selectize_widget.js | 2 +- .../widgets/jinja/synced_input_widget.js | 2 +- .../js/jquery/widgets/jinja/time_widget.js | 2 +- .../jquery/widgets/jinja/typeahead_widget.js | 2 +- indico/web/client/js/jquery/widgets/misc.js | 2 +- .../js/jquery/widgets/multitextfield.js | 2 +- .../js/jquery/widgets/nullableselector.js | 2 +- .../client/js/jquery/widgets/palettepicker.js | 2 +- .../widgets/paper_email_settings_widget.js | 2 +- .../js/jquery/widgets/principalfield.js | 24 +- .../web/client/js/jquery/widgets/qbubble.js | 2 +- .../js/jquery/widgets/realtimefilter.js | 2 +- .../js/jquery/widgets/rulelistwidget.js | 2 +- .../client/js/jquery/widgets/scrollblocker.js | 2 +- .../js/jquery/widgets/sticky_tooltip.js | 2 +- .../js/jquery/widgets/track_role_widget.js | 2 +- indico/web/client/js/legacy/angular/app.js | 2 +- .../client/js/legacy/angular/directives.js | 2 +- .../web/client/js/legacy/angular/filters.js | 2 +- indico/web/client/js/legacy/angular/index.js | 2 +- .../web/client/js/legacy/angular/services.js | 2 +- indico/web/client/js/legacy/indico.js | 4 +- .../js/legacy/libs/indico/Common/Export.js | 363 - .../js/legacy/libs/indico/Core/Auxiliar.js | 2 +- .../js/legacy/libs/indico/Core/Browser.js | 2 +- .../js/legacy/libs/indico/Core/Components.js | 4 +- .../client/js/legacy/libs/indico/Core/Data.js | 2 +- .../legacy/libs/indico/Core/Dialogs/Base.js | 2 +- .../legacy/libs/indico/Core/Dialogs/Popup.js | 2 +- .../legacy/libs/indico/Core/Dialogs/Users.js | 1602 +- .../legacy/libs/indico/Core/Dialogs/Util.js | 2 +- .../js/legacy/libs/indico/Core/Effects.js | 2 +- .../libs/indico/Core/Interaction/Base.js | 2 +- .../legacy/libs/indico/Core/Presentation.js | 2 +- .../client/js/legacy/libs/indico/Core/Util.js | 2 +- .../legacy/libs/indico/Core/Widgets/Base.js | 2 +- .../legacy/libs/indico/Core/Widgets/Inline.js | 82 +- .../legacy/libs/indico/Core/Widgets/Menu.js | 2 +- .../libs/indico/Core/Widgets/RichText.js | 2 +- .../js/legacy/libs/indico/Legacy/Util.js | 2 +- .../js/legacy/libs/indico/Legacy/Widgets.js | 2 +- .../legacy/libs/presentation/Core/Commands.js | 2 +- .../libs/presentation/Core/Interfaces.js | 2 +- .../libs/presentation/Core/Iterators.js | 2 +- .../legacy/libs/presentation/Core/MathEx.js | 2 +- .../libs/presentation/Core/Primitives.js | 2 +- .../legacy/libs/presentation/Core/String.js | 2 +- .../js/legacy/libs/presentation/Core/Tools.js | 2 +- .../js/legacy/libs/presentation/Core/Type.js | 2 +- .../js/legacy/libs/presentation/Data/Bag.js | 2 +- .../legacy/libs/presentation/Data/Binding.js | 2 +- .../legacy/libs/presentation/Data/DateTime.js | 2 +- .../js/legacy/libs/presentation/Data/Json.js | 2 +- .../js/legacy/libs/presentation/Data/Logic.js | 2 +- .../legacy/libs/presentation/Data/Remote.js | 127 - .../libs/presentation/Data/WatchList.js | 2 +- .../libs/presentation/Data/WatchObject.js | 19 +- .../libs/presentation/Data/WatchValue.js | 2 +- .../js/legacy/libs/presentation/Ui/Dom.js | 2 +- .../libs/presentation/Ui/Extensions/Layout.js | 2 +- .../js/legacy/libs/presentation/Ui/Html.js | 2 +- .../js/legacy/libs/presentation/Ui/Text.js | 2 +- .../presentation/Ui/Widgets/WidgetBase.js | 2 +- .../Ui/Widgets/WidgetComponents.js | 2 +- .../presentation/Ui/Widgets/WidgetControl.js | 2 +- .../legacy/libs/presentation/Ui/XElement.js | 2 +- .../js/legacy/libs/timetable/Actions.js | 16 +- .../client/js/legacy/libs/timetable/Base.js | 8 +- .../js/legacy/libs/timetable/DragAndDrop.js | 2 +- .../client/js/legacy/libs/timetable/Draw.js | 20 +- .../client/js/legacy/libs/timetable/Filter.js | 2 +- .../client/js/legacy/libs/timetable/Layout.js | 2 +- .../js/legacy/libs/timetable/Management.js | 12 +- .../client/js/legacy/libs/timetable/Undo.js | 2 +- indico/web/client/js/legacy/presentation.js | 3 +- indico/web/client/js/legacy/timetable.js | 2 +- indico/web/client/js/outdatedbrowser/index.js | 2 +- .../js/outdatedbrowser/outdatedbrowser.css | 2 +- .../js/outdatedbrowser/outdatedbrowser.js | 2 +- .../js/outdatedbrowser/supported-browsers.js | 2 +- .../client/js/react/components/Carousel.jsx | 2 +- .../js/react/components/Carousel.module.scss | 2 +- .../js/react/components/ClipboardButton.jsx | 7 +- .../js/react/components/EmailListField.jsx | 6 +- .../client/js/react/components/IButton.jsx | 22 +- .../js/react/components/ICSCalendarLink.jsx | 237 + .../components/ICSCalendarLink.module.scss | 49 + .../components/ManagementPageBackButton.jsx | 4 +- .../components/ManagementPageSubTitle.jsx | 4 +- .../react/components/ManagementPageTitle.jsx | 4 +- .../client/js/react/components/MathJax.jsx | 4 +- .../client/js/react/components/MessageBox.jsx | 4 +- .../web/client/js/react/components/Modal.jsx | 4 +- .../client/js/react/components/Paginator.jsx | 5 +- .../react/components/PopoverDropdownMenu.jsx | 6 +- .../PopoverDropdownMenu.module.scss | 2 +- .../js/react/components/RequestConfirm.jsx | 4 +- .../js/react/components/ResponsivePopup.jsx | 3 +- .../js/react/components/ReviewRating.jsx | 4 +- .../react/components/ReviewRating.module.scss | 2 +- .../js/react/components/ScrollButton.jsx | 6 +- .../react/components/ScrollButton.module.scss | 2 +- .../react/components/StickyWithScrollBack.jsx | 4 +- .../StickyWithScrollBack.module.scss | 2 +- .../react/components/TooltipIfTruncated.jsx | 4 +- .../client/js/react/components/UserMenu.jsx | 39 +- .../js/react/components/WTFDateField.jsx | 5 +- .../js/react/components/WTFDateTimeField.jsx | 11 +- .../react/components/WTFOccurrencesField.jsx | 9 +- .../js/react/components/WTFPrincipalField.jsx | 16 +- .../components/WTFPrincipalField.module.scss | 2 +- .../components/WTFPrincipalListField.jsx | 5 +- .../WTFPrincipalListField.module.scss | 2 +- .../js/react/components/WTFTimeField.jsx | 5 +- .../dates/CalendarRangeDatePicker.jsx | 4 +- .../dates/CalendarSingleDatePicker.jsx | 4 +- .../components/dates/DatePeriodField.jsx | 6 +- .../dates/DatePeriodField.module.scss | 2 +- .../components/dates/DateRangePicker.jsx | 3 +- .../components/dates/SingleDatePicker.jsx | 6 +- .../client/js/react/components/dates/util.jsx | 4 +- .../js/react/components/files/FileArea.jsx | 6 +- .../components/files/FileArea.module.scss | 2 +- .../react/components/files/FileSubmission.jsx | 4 +- .../components/files/SingleFileManager.jsx | 8 +- .../client/js/react/components/files/props.js | 2 +- .../client/js/react/components/files/util.js | 3 +- .../web/client/js/react/components/index.js | 19 +- .../react/components/principals/ACLField.jsx | 8 +- .../components/principals/PermissionTree.jsx | 16 +- .../principals/PermissionTree.module.scss | 2 +- .../components/principals/PrincipalField.jsx | 11 +- .../principals/PrincipalListField.jsx | 11 +- .../principals/PrincipalListField.module.scss | 2 +- .../principals/PrincipalPermissions.jsx | 16 +- .../PrincipalPermissions.module.scss | 2 +- .../js/react/components/principals/Search.jsx | 204 +- .../components/principals/Search.module.scss | 21 +- .../components/principals/TrackACLField.jsx | 6 +- .../js/react/components/principals/hooks.js | 13 +- .../react/components/principals/imperative.js | 57 + .../js/react/components/principals/index.js | 2 +- .../js/react/components/principals/items.jsx | 6 +- .../components/principals/items.module.scss | 2 +- .../js/react/components/principals/util.js | 10 +- .../js/react/components/style/dates.scss | 2 +- .../js/react/components/style/modal.scss | 2 +- .../js/react/components/style/paginator.scss | 2 +- .../client/js/react/containers/UserMenu.jsx | 3 +- indico/web/client/js/react/errors/actions.js | 2 +- .../web/client/js/react/errors/component.jsx | 11 +- .../web/client/js/react/errors/container.js | 4 +- indico/web/client/js/react/errors/index.jsx | 5 +- indico/web/client/js/react/errors/reducers.js | 2 +- indico/web/client/js/react/forms/errors.js | 4 +- indico/web/client/js/react/forms/fields.jsx | 6 +- .../client/js/react/forms/fields.module.scss | 2 +- .../web/client/js/react/forms/final-form.jsx | 8 +- .../web/client/js/react/forms/formatters.js | 2 +- indico/web/client/js/react/forms/index.js | 2 +- indico/web/client/js/react/forms/parsers.js | 2 +- indico/web/client/js/react/forms/unload.jsx | 7 +- .../web/client/js/react/forms/validators.js | 2 +- indico/web/client/js/react/hooks.js | 66 +- indico/web/client/js/react/i18n.js | 2 +- .../client/js/react/util/ConditionalRoute.jsx | 4 +- indico/web/client/js/react/util/Markdown.jsx | 4 +- indico/web/client/js/react/util/Preloader.jsx | 4 +- .../web/client/js/react/util/Responsive.jsx | 2 +- indico/web/client/js/react/util/Slot.jsx | 4 +- indico/web/client/js/react/util/html.js | 2 +- indico/web/client/js/react/util/index.js | 2 +- indico/web/client/js/react/util/propTypes.js | 2 +- indico/web/client/js/react/util/routing.js | 2 +- .../js/utils/__tests__/debounce.spec.js | 2 +- indico/web/client/js/utils/axios.js | 2 +- indico/web/client/js/utils/case.js | 2 +- indico/web/client/js/utils/date.js | 5 +- indico/web/client/js/utils/debounce.js | 2 +- indico/web/client/js/utils/i18n.js | 6 +- indico/web/client/js/utils/palette.js | 2 +- indico/web/client/js/utils/redux.js | 2 +- indico/web/client/styles/_base.scss | 2 +- indico/web/client/styles/_custom.scss | 2 +- indico/web/client/styles/_modules.scss | 2 +- indico/web/client/styles/_partials.scss | 2 +- indico/web/client/styles/_widgets.scss | 2 +- indico/web/client/styles/base/_animation.scss | 2 +- indico/web/client/styles/base/_borders.scss | 2 +- indico/web/client/styles/base/_defaults.scss | 2 +- indico/web/client/styles/base/_flex.scss | 2 +- indico/web/client/styles/base/_grid.scss | 2 +- .../web/client/styles/base/_indicators.scss | 2 +- indico/web/client/styles/base/_layout.scss | 3 +- indico/web/client/styles/base/_pages.scss | 2 +- indico/web/client/styles/base/_palette.scss | 2 +- indico/web/client/styles/base/_reset.scss | 2 +- indico/web/client/styles/base/_theme.scss | 2 +- .../web/client/styles/base/_typography.scss | 2 +- indico/web/client/styles/base/_utilities.scss | 2 +- .../web/client/styles/custom/_dropzone.scss | 2 +- .../styles/custom/_jquery-multiselect.scss | 2 +- .../styles/custom/_jquery-ui-datepicker.scss | 2 +- .../web/client/styles/custom/_jquery-ui.scss | 2 +- .../client/styles/custom/jquery-ui-style.css | 2 +- .../web/client/styles/legacy/Conf_Basic.scss | 2 +- indico/web/client/styles/legacy/Default.css | 51 +- indico/web/client/styles/legacy/timetable.css | 2 +- .../web/client/styles/modules/_abstracts.scss | 2 +- indico/web/client/styles/modules/_admin.scss | 2 +- .../client/styles/modules/_agreements.scss | 2 +- .../client/styles/modules/_attachments.scss | 2 +- indico/web/client/styles/modules/_auth.scss | 2 +- .../web/client/styles/modules/_bootstrap.scss | 2 +- .../web/client/styles/modules/_category.scss | 2 +- .../styles/modules/_category_management.scss | 2 +- .../client/styles/modules/_contributions.scss | 2 +- .../web/client/styles/modules/_dashboard.scss | 2 +- .../web/client/styles/modules/_designer.scss | 2 +- .../client/styles/modules/_event_display.scss | 2 +- .../styles/modules/_event_management.scss | 5 +- .../client/styles/modules/_eventservices.scss | 2 +- .../web/client/styles/modules/_markdown.scss | 2 +- indico/web/client/styles/modules/_news.scss | 2 +- .../web/client/styles/modules/_overviews.scss | 2 +- indico/web/client/styles/modules/_papers.scss | 2 +- .../web/client/styles/modules/_payment.scss | 2 +- .../styles/modules/_registrationform.scss | 14 +- .../web/client/styles/modules/_reviewing.scss | 9 +- indico/web/client/styles/modules/_roles.scss | 2 +- .../web/client/styles/modules/_sessions.scss | 2 +- .../web/client/styles/modules/_surveys.scss | 2 +- .../web/client/styles/modules/_timetable.scss | 2 +- indico/web/client/styles/modules/_tracks.scss | 2 +- .../client/styles/modules/_types_dialog.scss | 2 +- indico/web/client/styles/modules/_users.scss | 2 +- indico/web/client/styles/modules/_vc.scss | 2 +- .../modules/abstracts/_abstract_list.scss | 2 +- .../modules/abstracts/_abstract_page.scss | 2 +- .../styles/modules/abstracts/_cfa_page.scss | 2 +- .../modules/abstracts/_email_templates.scss | 2 +- .../modules/abstracts/_notification_logs.scss | 2 +- .../styles/modules/abstracts/_roles.scss | 9 +- .../modules/contributions/_display.scss | 2 +- .../styles/modules/contributions/_editor.scss | 2 +- .../styles/modules/event_display/_common.scss | 2 +- .../modules/event_display/_conferences.scss | 15 +- .../modules/event_display/_meetings.scss | 44 +- .../client/styles/partials/_actionboxes.scss | 2 +- .../web/client/styles/partials/_badges.scss | 8 +- indico/web/client/styles/partials/_boxes.scss | 2 +- .../web/client/styles/partials/_buttons.scss | 6 +- indico/web/client/styles/partials/_core.scss | 9 +- .../styles/partials/_corner-messages.scss | 2 +- .../web/client/styles/partials/_dialogs.scss | 2 +- .../client/styles/partials/_dropdowns.scss | 4 +- indico/web/client/styles/partials/_fonts.scss | 2 +- .../web/client/styles/partials/_footer.scss | 2 +- indico/web/client/styles/partials/_forms.scss | 28 +- indico/web/client/styles/partials/_icons.scss | 2 +- .../client/styles/partials/_infogrids.scss | 2 +- .../web/client/styles/partials/_inputs.scss | 2 +- .../_jquery-indico-categorynavigator.scss | 2 +- .../_jquery-indico-clearableinput.scss | 2 +- .../partials/_jquery-indico-colorpicker.scss | 2 +- .../partials/_jquery-indico-itempicker.scss | 2 +- .../_jquery-indico-locationpicker.scss | 2 +- .../_jquery-indico-multitextfield.scss | 2 +- .../_jquery-indico-palettepicker.scss | 2 +- .../partials/_jquery-indico-qbubble.scss | 2 +- .../partials/_jquery-indico-taglist.scss | 2 +- .../web/client/styles/partials/_labels.scss | 2 +- indico/web/client/styles/partials/_links.scss | 2 +- indico/web/client/styles/partials/_lists.scss | 2 +- indico/web/client/styles/partials/_main.scss | 2 +- .../client/styles/partials/_messageboxes.scss | 2 +- .../client/styles/partials/_object-lists.scss | 3 +- .../client/styles/partials/_paddedboxes.scss | 3 +- .../web/client/styles/partials/_payment.scss | 2 +- .../web/client/styles/partials/_plugins.scss | 2 +- .../web/client/styles/partials/_progress.scss | 2 +- indico/web/client/styles/partials/_qtips.scss | 2 +- .../web/client/styles/partials/_requests.scss | 2 +- indico/web/client/styles/partials/_rules.scss | 2 +- .../web/client/styles/partials/_sidebars.scss | 2 +- .../web/client/styles/partials/_spinner.scss | 2 +- indico/web/client/styles/partials/_steps.scss | 2 +- .../web/client/styles/partials/_switch.scss | 2 +- .../web/client/styles/partials/_tables.scss | 2 +- indico/web/client/styles/partials/_tags.scss | 2 +- .../client/styles/partials/_timelines.scss | 14 +- .../styles/partials/_timetable-balloons.scss | 2 +- .../web/client/styles/partials/_toolbars.scss | 6 +- .../web/client/styles/partials/_widgets.scss | 2 +- indico/web/client/styles/screen.scss | 2 +- indico/web/client/styles/themes/compact.scss | 2 +- indico/web/client/styles/themes/indico.scss | 3 +- indico/web/client/styles/themes/meeting.scss | 2 +- .../client/styles/themes/print/indico.scss | 2 +- indico/web/client/styles/themes/weeks.scss | 2 +- .../client/styles/widgets/_occurrences.scss | 2 +- .../client/styles/widgets/_permissions.scss | 2 +- .../client/styles/widgets/_person_link.scss | 2 +- .../client/styles/widgets/_sortable_list.scss | 2 +- indico/web/errors.py | 20 +- indico/web/fields/__init__.py | 6 +- indico/web/fields/base.py | 22 +- indico/web/fields/choices.py | 14 +- indico/web/fields/simple.py | 14 +- indico/web/flask/app.py | 136 +- indico/web/flask/errors.py | 27 +- indico/web/flask/session.py | 78 +- indico/web/flask/stats.py | 4 +- indico/web/flask/templating.py | 171 +- indico/web/flask/templating_test.py | 93 +- indico/web/flask/util.py | 48 +- indico/web/flask/util_test.py | 4 +- indico/web/flask/wrappers.py | 70 +- indico/web/forms/base.py | 66 +- indico/web/forms/colors.py | 6 +- indico/web/forms/fields/__init__.py | 3 +- indico/web/forms/fields/colors.py | 16 +- indico/web/forms/fields/datetime.py | 74 +- indico/web/forms/fields/enums.py | 16 +- indico/web/forms/fields/files.py | 10 +- indico/web/forms/fields/itemlists.py | 40 +- indico/web/forms/fields/location.py | 8 +- indico/web/forms/fields/markdown.py | 8 +- indico/web/forms/fields/principals.py | 24 +- indico/web/forms/fields/protection.py | 6 +- indico/web/forms/fields/simple.py | 24 +- indico/web/forms/fields/sqlalchemy.py | 10 +- indico/web/forms/fields/util.py | 5 +- indico/web/forms/jinja_helpers.py | 18 +- indico/web/forms/util.py | 11 +- indico/web/forms/validators.py | 90 +- indico/web/forms/widgets.py | 95 +- indico/web/http_api/__init__.py | 2 +- indico/web/http_api/exceptions.py | 3 +- indico/web/http_api/fossils.py | 53 - indico/web/http_api/handlers.py | 116 +- indico/web/http_api/hooks/base.py | 26 +- indico/web/http_api/hooks/file.py | 4 +- indico/web/http_api/metadata/__init__.py | 3 +- indico/web/http_api/metadata/atom.py | 8 +- indico/web/http_api/metadata/html.py | 2 +- indico/web/http_api/metadata/ical.py | 26 +- indico/web/http_api/metadata/json.py | 7 +- indico/web/http_api/metadata/jsonp.py | 8 +- indico/web/http_api/metadata/serializer.py | 9 +- indico/web/http_api/metadata/xml.py | 25 +- indico/web/http_api/responses.py | 78 +- indico/web/http_api/util.py | 61 +- indico/web/indico.wsgi | 2 +- indico/web/menu.py | 30 +- indico/web/rh.py | 92 +- indico/web/static/images/google_calendar.gif | Bin 0 -> 1167 bytes indico/web/static/images/robot.svg | 26 + indico/web/static/robots.txt | 12 +- indico/web/templates/_access_list.html | 23 +- indico/web/templates/_ical_export.html | 79 - .../web/templates/_protection_messages.html | 1 + indico/web/templates/_session_bar.html | 6 +- indico/web/templates/_sortable_list.html | 6 +- indico/web/templates/_statistics.html | 2 +- indico/web/templates/bad_url_error.html | 9 + indico/web/templates/error.html | 2 +- indico/web/templates/footer.html | 4 +- indico/web/templates/forms/_form.html | 10 +- .../forms/_person_link_widget_base.html | 6 +- .../forms/category_picker_widget.html | 4 - .../forms/checkbox_group_widget.html | 2 +- .../web/templates/forms/datetime_widget.html | 3 +- .../templates/forms/permissions_widget.html | 2 +- .../templates/forms/prefixed_text_widget.html | 9 + .../forms/principal_list_widget.html | 11 +- .../web/templates/forms/principal_widget.html | 3 +- .../templates/forms/radio_buttons_widget.html | 4 +- indico/web/templates/indico_base.html | 2 +- indico/web/templates/meta.html | 2 +- indico/web/templates/overview/base.html | 4 +- indico/web/templates/placeholder_info.html | 2 +- indico/web/templates/standalone_error.html | 2 +- indico/web/util.py | 267 +- indico/web/util_test.py | 534 +- indico/web/views.py | 44 +- jsconfig.json | 2 +- package-lock.json | 51749 ++++++++-------- package.json | 168 +- plugin.webpack.config.js | 8 +- postcss.config.js | 10 - pre-commit.githook | 34 +- pytest.ini | 6 +- requirements.dev.in | 23 + requirements.dev.txt | 258 +- requirements.in | 61 + requirements.txt | 359 +- setup.cfg | 43 + setup.py | 51 +- setupTests.js | 2 +- webpack.config.babel.js | 10 +- webpack/base.js | 72 +- webpack/index.js | 2 +- 1945 files changed, 99403 insertions(+), 49051 deletions(-) create mode 100644 .npmrc delete mode 100644 .travis.yml create mode 100644 SECURITY.md create mode 100644 docs/source/building/index.rst create mode 100644 headers.yml create mode 100644 indico/core/cache_test.py delete mode 100644 indico/core/celery/flower.py create mode 100644 indico/core/limiter.py create mode 100644 indico/core/oauth/__init__.py create mode 100644 indico/core/oauth/endpoints.py create mode 100644 indico/core/oauth/grants.py rename indico/{legacy/services/interface/rpc/handlers.py => core/oauth/logger.py} (50%) rename indico/{legacy/fossils => core/oauth/models}/__init__.py (100%) create mode 100644 indico/core/oauth/models/applications.py rename indico/{modules => core}/oauth/models/applications_test.py (83%) create mode 100644 indico/core/oauth/models/tokens.py create mode 100644 indico/core/oauth/models/tokens_test.py create mode 100644 indico/core/oauth/oauth2.py create mode 100644 indico/core/oauth/oauth2_test.py create mode 100644 indico/core/oauth/protector.py create mode 100644 indico/core/oauth/scopes.py rename indico/{legacy/services => core/oauth/testing}/__init__.py (100%) create mode 100644 indico/core/oauth/testing/fixtures.py create mode 100644 indico/core/oauth/util.py create mode 100644 indico/core/sentry.py delete mode 100644 indico/legacy/common/cache.py delete mode 100644 indico/legacy/common/contribPacker.py delete mode 100644 indico/legacy/fossils/user.py delete mode 100644 indico/legacy/services/implementation/base.py delete mode 100644 indico/legacy/services/implementation/search.py delete mode 100644 indico/legacy/services/interface/rpc/__init__.py delete mode 100644 indico/legacy/services/interface/rpc/json.py delete mode 100644 indico/legacy/services/interface/rpc/process.py delete mode 100644 indico/legacy/services/tools.py create mode 100644 indico/migrations/versions/20201103_1431_8d614ef75968_allow_mx_user_title.py create mode 100644 indico/migrations/versions/20201209_2010_e4fb983dc64c_add_until_approved_regform_modification_mode.py create mode 100644 indico/migrations/versions/20210129_2232_e787389ca868_add_rejection_reason.py create mode 100644 indico/migrations/versions/20210211_1613_26985db8ed12_add_attach_ical_to_reminders.py create mode 100644 indico/migrations/versions/20210215_1052_f26c201c8254_add_attach_ical_to_registrationform.py create mode 100644 indico/migrations/versions/20210219_1428_3782de7970da_rename_oauth_default_scopes.py create mode 100644 indico/migrations/versions/20210219_1555_da06d8f50342_separate_authorized_scopes_from_tokens.py create mode 100644 indico/migrations/versions/20210222_1754_c36abe1c23c7_make_oauth_pkce_flow_configurable.py create mode 100644 indico/migrations/versions/20210222_1914_d354278c6d95_store_tokens_as_hashes.py create mode 100644 indico/migrations/versions/20210224_1805_ecc7088914e7_use_cascading_fks_for_oauth.py create mode 100644 indico/migrations/versions/20210224_1808_26806768cd3f_remove_flower_oauth_app.py create mode 100644 indico/modules/attachments/client/js/legacy.js create mode 100644 indico/modules/attachments/client/js/package.js create mode 100644 indico/modules/attachments/controllers/display/event_test.py create mode 100644 indico/modules/attachments/tasks.py create mode 100644 indico/modules/categories/client/js/base.jsx create mode 100644 indico/modules/categories/controllers/util.py delete mode 100644 indico/modules/categories/templates/category_export_ical.html create mode 100644 indico/modules/categories/templates/management/_create_category_button.html create mode 100644 indico/modules/events/client/js/header.jsx create mode 100644 indico/modules/events/contributions/controllers/display_test.py create mode 100644 indico/modules/events/contributions/ical.py delete mode 100644 indico/modules/events/contributions/templates/display/contribution_ical_export.html create mode 100644 indico/modules/events/editing/client/js/editing/timeline/CustomActions.jsx rename indico/{web/client/js/react/components/UserMenu.module.scss => modules/events/editing/client/js/editing/timeline/CustomActions.module.scss} (65%) create mode 100644 indico/modules/events/ical.py create mode 100644 indico/modules/events/legacy_ids_test.py create mode 100644 indico/modules/events/persons/schemas.py create mode 100644 indico/modules/events/persons/templates/emails/_contributions.html create mode 100644 indico/modules/events/sessions/client/js/session_display.jsx create mode 100644 indico/modules/events/sessions/ical.py create mode 100644 indico/modules/events/sessions/schemas.py delete mode 100644 indico/modules/events/sessions/templates/display/session_ical_export.html delete mode 100644 indico/modules/events/templates/display/event_ical_export.html create mode 100644 indico/modules/events/timetable/controllers/display_test.py delete mode 100644 indico/modules/groups/legacy.py delete mode 100644 indico/modules/oauth/models/__init__.py delete mode 100644 indico/modules/oauth/models/applications.py delete mode 100644 indico/modules/oauth/models/tokens.py delete mode 100644 indico/modules/oauth/models/tokens_test.py delete mode 100644 indico/modules/oauth/provider.py delete mode 100644 indico/modules/oauth/provider_test.py delete mode 100644 indico/modules/oauth/testing/__init__.py delete mode 100644 indico/modules/oauth/testing/fixtures.py create mode 100644 indico/modules/users/client/js/Favorites.jsx create mode 100644 indico/modules/users/client/js/Favorites.module.scss delete mode 100644 indico/modules/users/client/js/ICSCalendarLink.jsx delete mode 100644 indico/modules/users/legacy.py delete mode 100644 indico/modules/users/templates/_favorites.html create mode 100644 indico/testing/fixtures/cache.py create mode 100644 indico/testing/fixtures/session.py create mode 100644 indico/translations/pl_PL/LC_MESSAGES/messages-js.po create mode 100644 indico/translations/pl_PL/LC_MESSAGES/messages-react.po create mode 100644 indico/translations/pl_PL/LC_MESSAGES/messages.po create mode 100644 indico/translations/uk_UA/LC_MESSAGES/messages-js.po create mode 100644 indico/translations/uk_UA/LC_MESSAGES/messages-react.po create mode 100644 indico/translations/uk_UA/LC_MESSAGES/messages.po delete mode 100644 indico/util/emails/__init__.py delete mode 100644 indico/util/emails/backend.py rename indico/util/{struct => }/enum.py (90%) delete mode 100644 indico/util/fossilize/__init__.py delete mode 100644 indico/util/fossilize/conversion.py rename indico/util/{struct => }/iterables.py (85%) create mode 100644 indico/util/iterables_test.py delete mode 100644 indico/util/struct/__init__.py delete mode 100644 indico/util/struct/iterables_test.py rename indico/{legacy/services/implementation => vendor}/__init__.py (100%) create mode 100644 indico/vendor/django_mail/__init__.py rename indico/{legacy/services/interface => vendor/django_mail/backends}/__init__.py (100%) create mode 100644 indico/vendor/django_mail/backends/base.py create mode 100644 indico/vendor/django_mail/backends/console.py create mode 100644 indico/vendor/django_mail/backends/locmem.py create mode 100644 indico/vendor/django_mail/backends/smtp.py create mode 100644 indico/vendor/django_mail/encoding_utils.py rename indico/{util/emails => vendor/django_mail}/message.py (59%) create mode 100644 indico/vendor/django_mail/module_loading_utils.py create mode 100644 indico/vendor/django_mail/utils.py create mode 100644 indico/web/args_test.py delete mode 100644 indico/web/client/js/legacy/libs/indico/Common/Export.js delete mode 100644 indico/web/client/js/legacy/libs/presentation/Data/Remote.js create mode 100644 indico/web/client/js/react/components/ICSCalendarLink.jsx create mode 100644 indico/web/client/js/react/components/ICSCalendarLink.module.scss create mode 100644 indico/web/client/js/react/components/principals/imperative.js delete mode 100644 indico/web/http_api/fossils.py create mode 100644 indico/web/static/images/google_calendar.gif create mode 100644 indico/web/static/images/robot.svg delete mode 100644 indico/web/templates/_ical_export.html create mode 100644 indico/web/templates/bad_url_error.html create mode 100644 indico/web/templates/forms/prefixed_text_widget.html delete mode 100644 postcss.config.js create mode 100644 requirements.dev.in create mode 100644 requirements.in diff --git a/.coveragerc b/.coveragerc index b1dd494129b..972c02e24cf 100644 --- a/.coveragerc +++ b/.coveragerc @@ -2,7 +2,6 @@ branch = true source = indico omit = - indico/core/fossils/* indico/legacy/* indico/migrations/* indico/testing/* diff --git a/.eslintrc.yml b/.eslintrc.yml index 6b666a683ee..cf66b6bfacd 100644 --- a/.eslintrc.yml +++ b/.eslintrc.yml @@ -15,8 +15,6 @@ extends: - 'indico/react' - 'indico/react-hooks' - 'indico/prettier' - - 'prettier/babel' - - 'prettier/react' settings: # we don't use the webpack resolver because it is SLOW (~1s), @@ -30,9 +28,10 @@ settings: - ['indico/modules/users', './indico/modules/users/client/js'] - ['indico/modules/events/reviewing', './indico/modules/events/client/js/reviewing'] - ['indico/modules/events/editing', './indico/modules/events/editing/client/js'] - - ['indico/modules/events/util', './indico/modules/events/client/js/util'] + - ['indico/modules/events', './indico/modules/events/client/js'] - ['indico', './indico/web/client/js'] extensions: [.js, .jsx, .json] + import/internal-regex: ^indico/ react: version: detect @@ -45,6 +44,21 @@ rules: - ignore: ['^indico-url:'] import/no-cycle: - warn + import/order: + - error + - groups: [builtin, external, internal, parent, sibling, index] + alphabetize: + order: asc + caseInsensitive: true + pathGroups: + - pattern: indico-url:* + group: external + position: before + - pattern: '{.,..,../..,../../..}/**/*.+(css|scss)' # eslint-plugin-import#1239 + group: sibling + position: after + pathGroupsExcludedImportTypes: [builtin] + newlines-between: always new-cap: - error - capIsNewExceptionPattern: '\$\.(Event|Deferred)$' diff --git a/.flake8 b/.flake8 index 0971472f9f1..1ca3216038e 100644 --- a/.flake8 +++ b/.flake8 @@ -19,6 +19,7 @@ exclude = indico.egg-info node_modules .*/ + indico.conf # TODO: remove the next two entries and use extend-exclude once flake8 3.8.0 is out .git __pycache__ @@ -45,3 +46,5 @@ per-file-ignores = indico/util/mdx_latex_test.py:E501 # allow nicely aligned parametrizations indico/*_test.py:E241 + # sphinx config has many commented-out variables with no space after `#` + docs/source/conf.py:E265 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c99ba5152ab..649e51ef9f8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,22 +2,20 @@ name: CI on: push: - branches: [master, actions] + branches: + - master + - 2.3-maintenance pull_request: - branches: [master, actions] + branches: + - master + - 2.3-maintenance jobs: setup: - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 - - uses: actions/cache@v2 - id: cache-pip - with: - path: .venv - key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }} - - uses: actions/cache@v2 id: cache-npm with: @@ -25,27 +23,42 @@ jobs: key: ${{ runner.os }}-npm-${{ hashFiles('package*.json') }} - name: Setup Python - uses: actions/setup-python@v1 - if: steps.cache-pip.outputs.cache-hit != 'true' + uses: actions/setup-python@v2 with: - python-version: 2.7 + python-version: 3.9 - name: Setup Node uses: actions/setup-node@v1 if: steps.cache-npm.outputs.cache-hit != 'true' with: - node-version: 12.x + node-version: 14.x - name: Install system dependencies - if: steps.cache-pip.outputs.cache-hit != 'true' - run: sudo apt-get install libpq-dev python-virtualenv + run: sudo apt-get install libpq-dev - - name: Install python dependencies - if: steps.cache-pip.outputs.cache-hit != 'true' + - name: Create virtualenv run: | - virtualenv -p /usr/bin/python2.7 .venv + python3.9 -m venv .venv source .venv/bin/activate - pip install -U pip setuptools + pip install -U pip setuptools wheel + + - name: Activate virtualenv for later steps + run: | + echo "VIRTUAL_ENV=$(pwd)/.venv" >> $GITHUB_ENV + echo "$(pwd)/.venv/bin" >> $GITHUB_PATH + + - name: Get pip cache dir + id: pip-cache + run: echo "::set-output name=dir::$(pip cache dir)" + + - name: Cache pip + uses: actions/cache@v2 + with: + path: ${{ steps.pip-cache.outputs.dir }} + key: pip|${{ runner.os }}|3.9|${{ hashFiles('requirements*.txt') }} + + - name: Install python dependencies + run: | pip install -r requirements.dev.txt pip install -r requirements.txt @@ -53,47 +66,54 @@ jobs: if: steps.cache-npm.outputs.cache-hit != 'true' run: npm ci + - name: Archive environment + run: tar cf /tmp/env.tar .venv node_modules + + - name: Upload environment + uses: actions/upload-artifact@v2 + with: + name: environment + retention-days: 1 + path: /tmp/env.tar + lint: needs: setup - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: # BEGIN common steps - edit all occurrences if needed! - uses: actions/checkout@v2 - - uses: actions/cache@v2 - id: cache-pip - with: - path: .venv - key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }} - - - uses: actions/cache@v2 - id: cache-npm - with: - path: node_modules - key: ${{ runner.os }}-npm-${{ hashFiles('package*.json') }} - - name: Setup Python - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: - python-version: 2.7 + python-version: 3.9 - name: Setup Node uses: actions/setup-node@v1 with: - node-version: 12.x + node-version: 14.x + + - name: Download environment + uses: actions/download-artifact@v2 + with: + name: environment + path: /tmp + + - name: Restore environment + run: tar xf /tmp/env.tar - name: Activate virtualenv for later steps run: | - echo "::set-env name=VIRTUAL_ENV::$(pwd)/.venv" - echo "::add-path::$(pwd)/.venv/bin" + echo "VIRTUAL_ENV=$(pwd)/.venv" >> $GITHUB_ENV + echo "$(pwd)/.venv/bin" >> $GITHUB_PATH - name: Install Indico run: pip install -e . # END common steps - name: Check import sorting - run: isort -rc --diff --check-only indico/ + run: isort --diff --check-only indico/ - name: Check backref comments if: success() || failure() @@ -103,7 +123,7 @@ jobs: if: success() || failure() run: | echo "::add-matcher::.github/matchers/headers-problem-matcher.json" - python bin/maintenance/update_header.py indico --ci + python bin/maintenance/update_header.py --ci echo "::remove-matcher owner=headers::" - name: Run flake8 @@ -126,6 +146,8 @@ jobs: indico/modules/rb/ indico/modules/events/logs/ indico/modules/events/editing/ + indico/modules/events/client/js/reviewing/ + indico/modules/events/papers/client/js/ indico/web/client/js/react/ indico/modules/users/ @@ -157,7 +179,7 @@ jobs: test-python: needs: setup - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 services: postgres: @@ -172,37 +194,37 @@ jobs: # BEGIN common steps - edit all occurrences if needed! - uses: actions/checkout@v2 - - uses: actions/cache@v2 - id: cache-pip - with: - path: .venv - key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }} - - - uses: actions/cache@v2 - id: cache-npm - with: - path: node_modules - key: ${{ runner.os }}-npm-${{ hashFiles('package*.json') }} - - name: Setup Python - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: - python-version: 2.7 + python-version: 3.9 - name: Setup Node uses: actions/setup-node@v1 with: - node-version: 12.x + node-version: 14.x + + - name: Download environment + uses: actions/download-artifact@v2 + with: + name: environment + path: /tmp + + - name: Restore environment + run: tar xf /tmp/env.tar - name: Activate virtualenv for later steps run: | - echo "::set-env name=VIRTUAL_ENV::$(pwd)/.venv" - echo "::add-path::$(pwd)/.venv/bin" + echo "VIRTUAL_ENV=$(pwd)/.venv" >> $GITHUB_ENV + echo "$(pwd)/.venv/bin" >> $GITHUB_PATH - name: Install Indico run: pip install -e . # END common steps + - name: Install redis + run: sudo apt-get install redis-server + - name: Setup database run: | sudo apt-get install postgresql-client libpq-dev @@ -222,44 +244,42 @@ jobs: test-js: needs: setup - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: # BEGIN common steps - edit all occurrences if needed! - uses: actions/checkout@v2 - - uses: actions/cache@v2 - id: cache-pip - with: - path: .venv - key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }} - - - uses: actions/cache@v2 - id: cache-npm - with: - path: node_modules - key: ${{ runner.os }}-npm-${{ hashFiles('package*.json') }} - - name: Setup Python - uses: actions/setup-python@v1 + uses: actions/setup-python@v2 with: - python-version: 2.7 + python-version: 3.9 - name: Setup Node uses: actions/setup-node@v1 with: - node-version: 12.x + node-version: 14.x + + - name: Download environment + uses: actions/download-artifact@v2 + with: + name: environment + path: /tmp + + - name: Restore environment + run: tar xf /tmp/env.tar - name: Activate virtualenv for later steps run: | - echo "::set-env name=VIRTUAL_ENV::$(pwd)/.venv" - echo "::add-path::$(pwd)/.venv/bin" + echo "VIRTUAL_ENV=$(pwd)/.venv" >> $GITHUB_ENV + echo "$(pwd)/.venv/bin" >> $GITHUB_PATH - name: Install Indico run: pip install -e . # END common steps - - name: Run jest tests - run: npm test + # XXX: enzyme is not yet compatible with react 17 - https://github.com/enzymejs/enzyme/issues/2429 + # - name: Run jest tests + # run: npm test - name: Try building assets run: python bin/maintenance/build-assets.py indico --dev diff --git a/.gitignore b/.gitignore index a4625a67f37..fe6f6847d7d 100644 --- a/.gitignore +++ b/.gitignore @@ -32,6 +32,7 @@ node_modules # testing .coverage +.coverage.* coverage.xml htmlcov/ /.cache/ diff --git a/.isort.cfg b/.isort.cfg index 458afaf998b..d780987cc4b 100644 --- a/.isort.cfg +++ b/.isort.cfg @@ -3,7 +3,6 @@ line_length=120 multi_line_output=0 lines_after_imports=2 sections=FUTURE,STDLIB,THIRDPARTY,INDICO,FIRSTPARTY,LOCALFOLDER -known_third_party=flask_multipass,flask_pluginengine,flower +known_third_party=flask_multipass,flask_pluginengine known_indico=indico skip_glob=20??????????_*_*.py -not_skip=__init__.py diff --git a/.npmrc b/.npmrc new file mode 100644 index 00000000000..521a9f7c077 --- /dev/null +++ b/.npmrc @@ -0,0 +1 @@ +legacy-peer-deps=true diff --git a/.readthedocs.yml b/.readthedocs.yml index fc5b63f27a2..3b421fd3419 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -1,5 +1,7 @@ requirements_file: docs/requirements.txt +build: + image: testing python: - version: 2.7 + version: 3.9 setup_py_install: false pip_install: true diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 15129010af8..00000000000 --- a/.travis.yml +++ /dev/null @@ -1,39 +0,0 @@ -language: python -python: - - 2.7 -git: - submodules: false - depth: 500 -branches: - only: - - 2.1-maintenance - - 2.2-maintenance -addons: - postgresql: '9.6' -env: - - NODE_VERSION="12.16.1" -before_install: - - nvm install $NODE_VERSION -install: - - pip install -U pip setuptools - - pip install -r requirements.dev.txt - - pip install -e . - - npm ci -script: - - FORCE_COLOR=1 npx react-jsx-i18n extract --ext jsx indico/web/client/ indico/modules/ > /dev/null - - isort -rc --diff --check-only indico/ - - python bin/maintenance/update_backrefs.py --ci - - python bin/maintenance/update_header.py indico --ci - - flake8 - - npx eslint --ext .js --ext .jsx indico/modules/rb/ indico/modules/events/logs/ indico/web/client/js/react/ - indico/modules/users/ - - pytest - - npm test - - python bin/maintenance/build-assets.py indico --dev -notifications: - email: false - irc: - channels: - - 'chat.freenode.net#indico' - use_notice: true - skip_join: true diff --git a/.watchmanconfig b/.watchmanconfig index 189731aed25..c94bdec8a2f 100644 --- a/.watchmanconfig +++ b/.watchmanconfig @@ -1,3 +1,3 @@ { - "ignore_dirs": ["node_modules", "build"] + "ignore_dirs": ["node_modules", "build", "dist"] } diff --git a/CHANGES.rst b/CHANGES.rst index 3160dc0152a..20c78500185 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -2,16 +2,296 @@ Changelog ========= -Version 2.3.1 +Version 3.0 +----------- + +*Unreleased* + +Major Features +^^^^^^^^^^^^^^ + +- The OAuth provider module has been re-implemented based on a more modern + library (authlib). Support for the somewhat insecure *implicit flow* has been + removed in favor of the code-with-PKCE flow. Tokens are now stored more securely + as a hash instead of plaintext. For a given user/app/scope combination, only a + certain amount of tokens are stored; once the limit has been reached older tokens + will be discarded. The OAuth provider now exposes its metadata via a well-known + URI (RFC 8414) and also has endpoints to introspect or revoke a token. (:issue:`4685`, + :pr:`4798`) + +Improvements +^^^^^^^^^^^^ + +- Categories may now contain both events and subcategories at the same time + (:issue:`4679`, :pr:`4725`, :pr:`4757`) +- Show the user's profile picture in many more places (:issue:`4625`, :pr:`4747`) +- Use a more modern search dialog when searching for users (:issue:`4674`, :pr:`4743`) +- Add an option to refresh event person data from the underlying user when cloning an + event (:issue:`4750`, :pr:`4760`) +- Add options for attaching iCal files to complete registration and event reminder + emails (:issue:`1158`, :pr:`4780`) +- Use the new token-based URLs instead of API keys for persistent ical links and replace + the calendar link widgets in category, event, session and contribution views with the + more modern ones used in dashboard (:issue:`4776`, :pr:`4801`) +- Add an option to export editables to JSON (:issue:`4767`, :pr:`4810`) +- Add an option to export paper peer reviewing data to JSON (:issue:`4767`, :pr:`4818`) +- Passwords are now checked against a list of breached passwords ("Have I Been Pwned") + in a secure and anonymous way that does not disclose any data. If a user logs in with + an insecure password, they are forced to change it before they can continue using Indico + (:pr:`4817`) +- Failed login attempts now trigger rate limiting to prevent brute-force attacks + (:issue:`1550`, :pr:`4817`) +- Allow filtering the "Participant Roles" page by users who have not registered for the event + (:issue:`4763`, :pr:`4822`) +- iCalendar exports now include contact data, event logo URL and, when exporting + sessions/contributions, the UID of the related event. Also, only non-empty fields + are exported. (:issue:`4785`, :issue:`4586`, :issue:`4587`, :issue:`4791`, + :pr:`4820`) +- Allow adding groups/roles as "authorized abstract submitters" (:pr:`4834`) +- Direct links to (sub-)contributions in meetings using the URLs usually meant for + conferences now redirect to the meeting view page (:pr:`4847`) +- Use a more compact setup QR code for the mobile *Indico check-in* app; the latest version of + the app is now required. (:pr:`4844`) + +Bugfixes +^^^^^^^^ + +- Take registrations of users who are only members of a custom event role into account on the + "Participant Roles" page (:pr:`4822`) +- Fail gracefully during registration import when two rows have different emails that belong + to the same user (:pr:`4823`) +- Restore the ability to see who's inheriting access from a parent object (:pr:`4833`) +- Fix misleading message when cancelling a booking that already started and has past + occurrences that won't be cancelled (:issue:`4719`, :pr:`4861`) + +Internal Changes +^^^^^^^^^^^^^^^^ + +- Require Python 3.9 - older Python versions (especially Python 2.7) are **no longer supported** +- ``confId`` has been changed to ``event_id`` and the corresponding URL path segments + now enforce numeric data (and thus pass the id as a number instead of string) +- ``CACHE_BACKEND`` has been removed; Indico now always uses Redis for caching +- The integration with flower (celery monitoring tool) has been removed as it was not widely used, + did not provide much benefit, and it is no longer compatible with the latest Celery version +- ``session.user`` now returns the user related to the current request, regardless of whether + it's coming from OAuth, a signed url or the actual session (:pr:`4803`) +- Add a new ``check_password_secure`` signal that can be used to implement additional password + security checks (:pr:`4817`) + + +---- + +Version 2.3.5 ------------- *Unreleased* +Internationalization +^^^^^^^^^^^^^^^^^^^^ + +- New translation: Polish + +Improvements +^^^^^^^^^^^^ + +- Add an option to not disclose the names of editors and commenters to submitters in the + Paper Editing module (:issue:`4829`, :pr:`4865`) + +Bugfixes +^^^^^^^^ + +- Do not show soft-deleted long-lasting events in category calendar (:pr:`4824`) +- Do not show management-related links in editing hybrid view unless the user has + access to them (:pr:`4830`) +- Fix error when assigning paper reviewer roles with notifications enabled and one + of the reviewing types disabled (:pr:`4838`) +- Fix viewing timetable entries if you cannot access the event but a specific session + inside it (:pr:`4857`) +- Fix viewing contributions if you cannot access the event but have explicit access to + the contribution (:pr:`4860`) +- Hide registration menu item if you cannot access the event and registrations are not + exempt from event access checks (:pr:`4860`) +- Fix inadvertently deleting a file uploaded during the "make changes" Editing action, + resulting in the revision sometimes still referencing the file even though it has been + deleted from storage (:pr:`4866`) + +Version 2.3.4 +------------- + +*Released on March 11, 2021* + +Security fixes +^^^^^^^^^^^^^^ + +- Fix some open redirects which could help making harmful URLs look more trustworthy by linking + to Indico and having it redirect the user to a malicious site (:issue:`4814`, :pr:`4815`) +- The :data:`BASE_URL` is now always enforced and requests whose Host header does not match + are rejected. This prevents malicious actors from tricking Indico into sending e.g. a + password reset link to a user that points to a host controlled by the attacker instead of + the actual Indico host (:pr:`4815`) + +.. note:: + + If the webserver is already configured to enforce a canonical host name and redirects or + rejects such requests, this cannot be exploited. Additionally, exploiting this problem requires + user interaction: they would need to click on a password reset link which they never requested, + and which points to a domain that does not match the one where Indico is running. + +Improvements +^^^^^^^^^^^^ + +- Fail more gracefully is a user has an invalid locale set and fall back to the default + locale or English in case the default locale is invalid as well +- Log an error if the configured default locale does not exist +- Add ID-1 page size for badge printing (:pr:`4774`, thanks :user:`omegak`) +- Allow managers to specify a reason when rejecting registrants and add a new placeholder + for the rejection reason when emailing registrants (:pr:`4769`, thanks :user:`vasantvohra`) + +Bugfixes +^^^^^^^^ + +- Fix the "Videoconference Rooms" page in conference events when there are any VC rooms + attached but the corresponding plugin is no longer installed +- Fix deleting events which have a videoconference room attached which has its VC plugin + no longer installed +- Do not auto-redirect to SSO when an MS office user agent is detected (:issue:`4720`, + :pr:`4731`) +- Allow Editing team to view editables of unpublished contributions (:issue:`4811`, :pr:`4812`) + +Internal Changes +^^^^^^^^^^^^^^^^ + +- Also trigger the ``ical-export`` metadata signal when exporting events for a whole category +- Add ``primary_email_changed`` signal (:pr:`4802`, thanks :user:`openprojects`) + +Version 2.3.3 +------------- + +*Released on January 25, 2021* + +Security fixes +^^^^^^^^^^^^^^ + +- JSON locale data for invalid locales is no longer cached on disk; instead a 404 error is + triggered. This avoids creating small files in the cache folder for each invalid locale + that is requested. (:pr:`4766`) + +Internationalization +^^^^^^^^^^^^^^^^^^^^ + +- New translation: Ukrainian + +Improvements +^^^^^^^^^^^^ + +- Add a new "Until approved" option for a registration form's "Modification allowed" + setting (:pr:`4740`, thanks :user:`vasantvohra`) +- Show last login time in dashboard (:pr:`4735`, thanks :user:`vasantvohra`) +- Allow Markdown in the "Message for complete registrations" option of a registration + form (:pr:`4741`) +- Improve video conference linking dropdown for contributions/sessions (hide unscheduled, + show start time) (:pr:`4753`) +- Show timetable filter button in conferences with a meeting-like timetable + +Bugfixes +^^^^^^^^ + +- Fix error when converting malformed HTML links to LaTeX +- Hide inactive contribution/abstract fields in submit/edit forms (:pr:`4755`) +- Fix adding registrants to a session ACL + +Internal Changes +^^^^^^^^^^^^^^^^ + +- Videoconference plugins may now display a custom message for the prompt when deleting + a videoconference room (:pr:`4733`) +- Videoconference plugins may now override the behavior when cloning an event with + attached videoconference rooms (:pr:`4732`) + +Version 2.3.2 +------------- + +*Released on November 30, 2020* + +Improvements +^^^^^^^^^^^^ + +- Disable title field by default in new registration forms (:issue:`4688`, :pr:`4692`) +- Add gender-neutral "Mx" title (:issue:`4688`, :pr:`4692`) +- Add contributions placeholder for emails (:pr:`4716`, thanks :user:`bpedersen2`) +- Show program codes in contribution list (:pr:`4713`) +- Display the target URL of link materials if the user can access them (:issue:`2599`, + :pr:`4718`) +- Show the revision number for all revisions in the Editing timeline (:pr:`4708`) + +Bugfixes +^^^^^^^^ + +- Only consider actual speakers in the "has registered speakers" contribution list filter + (:pr:`4712`, thanks :user:`bpedersen2`) +- Correctly filter events in "Sync with your calendar" links (this fix only applies to newly + generated links) (:pr:`4717`) +- Correctly grant access to attachments inside public sessions/contribs even if the event + is more restricted (:pr:`4721`) +- Fix missing filename pattern check when suggesting files from Paper Peer Reviewing to submit + for Editing (:pr:`4715`) +- Fix filename pattern check in Editing when a filename contains dots (:pr:`4715`) +- Require explicit admin override (or being whitelisted) to override blockings (:pr:`4706`) +- Clone custom abstract/contribution fields when cloning abstract settings (:pr:`4724`, + thanks :user:`bpedersen2`) +- Fix error when rescheduling a survey that already has submissions (:issue:`4730`) + +Version 2.3.1 +------------- + +*Released on October 27, 2020* + +Security fixes +^^^^^^^^^^^^^^ +- Fix potential data leakage between OAuth-authenticated and unauthenticated HTTP API requests + for the same resource (:pr:`4663`) + +.. note:: + + Due to OAuth access to the HTTP API having been broken until this version, we do not + believe this was actually exploitable on any Indico instance. In addition, only Indico + administrators can create OAuth applications, so regardless of the bug there is no risk + for any instance which does not have OAuth applications with the ``read:legacy_api`` + scope. + +Improvements +^^^^^^^^^^^^ + +- Generate material packages in a background task to avoid timeouts or using excessive + amounts of disk space in case of people submitting several times (:pr:`4630`) +- Add new :data:`EXPERIMENTAL_EDITING_SERVICE` setting to enable extending an event's Editing + workflow through an `OpenReferee server `_ (:pr:`4659`) + Bugfixes ^^^^^^^^ - Only show the warning about draft mode in a conference if it actually has any contributions or timetable entries +- Do not show incorrect modification deadline in abstract management area if no + such deadline has been set (:pr:`4650`) +- Fix layout problem when minutes contain overly large embedded images (:issue:`4653`, + :pr:`4654`) +- Prevent pending registrations from being marked as checked-in (:pr:`4646`, thanks + :user:`omegak`) +- Fix OAuth access to HTTP API (:pr:`4663`) +- Fix ICS export of events with draft timetable and contribution detail level + (:pr:`4666`) +- Fix paper revision submission field being displayed for judges/reviewers (:pr:`4667`) +- Fix managers not being able to submit paper revisions on behalf of the user (:pr:`4667`) + +Internal Changes +^^^^^^^^^^^^^^^^ + +- Add ``registration_form_wtform_created`` signal and send form data in + ``registration_created`` and ``registration_updated`` signals (:pr:`4642`, + thanks :user:`omegak`) +- Add ``logged_in`` signal + Version 2.3 ----------- diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 2ba0c98b77c..ad2503218df 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,2 +1,2 @@ -Indico is a CERN project and is thus subject to [CERN's Code of Conduct](https://hr-dep.web.cern.ch/content/code-of-conduct). Outside contributors are expected to act in accordance with the principles of the CoC that apply to them. -The development team commits to ensuring that [CERN's values](https://hr-dep.web.cern.ch/content/cern-values-0) are respected and reserves the right to suspend from the community any contributors that are found to be engaged in harmful behaviour. Personal attacks or any other form of harassment will not be tolerated. +Indico is a CERN project and is thus subject to [CERN's Code of Conduct](https://hr.web.cern.ch/codeofconduct). Outside contributors are expected to act in accordance with the principles of the CoC that apply to them. +The development team commits to ensuring that [CERN's values](https://hr.web.cern.ch/cerns-values) are respected and reserves the right to suspend from the community any contributors that are found to be engaged in harmful behaviour. Personal attacks or any other form of harassment will not be tolerated. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dfa89b8e3fa..a2723fb3a41 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -65,6 +65,6 @@ This checklist is meant to help you follow those practices: ## Code of Conduct -Indico is a CERN project and is thus subject to [CERN's Code of Conduct](https://hr-dep.web.cern.ch/content/code-of-conduct). Outside contributors are expected to act in accordance with the principles of the CoC that apply to them. -The development team commits to ensuring that [CERN's values](https://hr-dep.web.cern.ch/content/cern-values-0) are respected and reserves the right to suspend from the community any contributors that are found to be engaged in harmful behaviour. Personal attacks or any other form of harassment will not be tolerated. +Indico is a CERN project and is thus subject to [CERN's Code of Conduct](https://hr.web.cern.ch/codeofconduct). Outside contributors are expected to act in accordance with the principles of the CoC that apply to them. +The development team commits to ensuring that [CERN's values](https://hr.web.cern.ch/cerns-values) are respected and reserves the right to suspend from the community any contributors that are found to be engaged in harmful behaviour. Personal attacks or any other form of harassment will not be tolerated. diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index a61ab27622c..f8421e2ba51 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -31,3 +31,12 @@ While keeping it would not hurt, it's better to stay in sync with the SQLAlchemy When writing/changing models or alembic revisions, run `python bin/maintenance/update_backrefs.py` to keep comments about relationship backrefs in sync and `python bin/utils/db_diff.py` to compare the models against your current database both to ensure your alembic revision is correct and that your own database is up to date. + + +## Updating Python dependencies +We use [pip-tools](https://github.com/jazzband/pip-tools) to manage our requirements.txt files. To update the pinned +dependencies, use `pip-compile -U` for regular dependencies and then `pip-compile -U requirements.dev.in` for dev +dependencies. Afterwards, check the diff for the requirements.txt files and consult each package's changelog for +important changes in case of direct dependencies that aren't just patch releases. Once that's done, you MUST install +Indico with `pip install -e '.[dev]'` and ensure nothing is broken (depending on what changed, make sure to test affected +parts manually). diff --git a/MANIFEST.in b/MANIFEST.in index a5fb12339bd..85f451955ae 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,5 +1,5 @@ include indico/*.sample indico/web/indico.wsgi -recursive-include indico *.html *.tpl *.txt *.js *.yaml *.tex *.svg +recursive-include indico *.html *.txt *.js *.yaml *.tex *.svg exclude indico/logging.yaml graft indico/core/plugins/alembic/ diff --git a/README.md b/README.md index 0a3e5422794..a2e10c661d7 100644 --- a/README.md +++ b/README.md @@ -71,8 +71,8 @@ The main meeting points for the community are: * the Chat Room ([#indico on Freenode](https://webchat.freenode.net/?channels=indico) or on [Matrix](https://riot.im/app/#/room/#indico:matrix.org)). -We follow [CERN's Values](https://hr-dep.web.cern.ch/content/cern-values-0) and the principles established by -[CERN's Code of Conduct](https://hr-dep.web.cern.ch/content/code-of-conduct). +We follow [CERN's Values](https://hr.web.cern.ch/cerns-values) and the principles established by +[CERN's Code of Conduct](https://hr.web.cern.ch/codeofconduct). ## History diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000000..4f985e95e93 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,25 @@ +# Security Policy + +## Supported Versions + +Indico uses the second part of the version number for major feature releases, ie. +2.2, 2.3, ... + +**Bugfixes are generally only released for the latest major version (e.g. 2.3.1 to +fix bugs discovered in 2.3).** + +**For security releases we usually follow the same schema.** In exceptional cases +where the previous version (e.g. 2.2) is still somewhat recent and thus widely +used AND no suitable workarounds exist, we may also create a patch release for +that version. + +Once version 3.0 (currently under development) is out, the same versioning +schema will apply there. However, due to the move from Python 2 or Python 3 and +the former having reached its end-of-live, no more 2.3.x releases will be made +after that point, and anyone running an Indico instance will be expected to move +to 3.0 as security cannot be guaranteed in an environment that has reached its +end-of-life. + +## Reporting a Vulnerability + +Please email indico-team@cern.ch to report security vulnerabilities privately. diff --git a/__mocks__/axios.js b/__mocks__/axios.js index 335f0fa4169..72212b71a6e 100644 --- a/__mocks__/axios.js +++ b/__mocks__/axios.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/__mocks__/react.js b/__mocks__/react.js index 80261984651..6e103d4c012 100644 --- a/__mocks__/react.js +++ b/__mocks__/react.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/babel.config.js b/babel.config.js index 469d399d9c9..bc1d5b6f91f 100644 --- a/babel.config.js +++ b/babel.config.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -16,7 +16,7 @@ const plugins = [ '@babel/plugin-proposal-class-properties', 'lodash', [ - 'react-css-modules', + '@dr.pogodin/react-css-modules', { exclude: 'node_modules', context: 'indico/modules', @@ -26,6 +26,7 @@ const plugins = [ }, }, autoResolveMultipleImports: true, + generateScopedName: '[path]___[name]__[local]___[hash:base64:5]', }, ], 'macros', diff --git a/bin/maintenance/build-assets.py b/bin/maintenance/build-assets.py index ce8179fd42f..06dc0d25a2d 100755 --- a/bin/maintenance/build-assets.py +++ b/bin/maintenance/build-assets.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -55,7 +55,7 @@ def _get_webpack_build_config(url_root='/'): 'distURL': os.path.join(url_root, 'dist/') }, 'themes': {key: {'stylesheet': theme['stylesheet'], 'print_stylesheet': theme.get('print_stylesheet')} - for key, theme in themes['definitions'].viewitems() + for key, theme in themes['definitions'].items() if set(theme) & {'stylesheet', 'print_stylesheet'}} } @@ -64,7 +64,7 @@ def _get_plugin_bundle_config(plugin_dir): try: with open(os.path.join(plugin_dir, 'webpack-bundles.json')) as f: return json.load(f) - except IOError as e: + except OSError as e: if e.errno == errno.ENOENT: return {} raise @@ -74,7 +74,7 @@ def _get_plugin_build_deps(plugin_dir): try: with open(os.path.join(plugin_dir, 'required-build-plugins.json')) as f: return json.load(f) - except IOError as e: + except OSError as e: if e.errno == errno.ENOENT: return [] raise @@ -86,10 +86,10 @@ def _parse_plugin_theme_yaml(plugin_yaml): core_data = f.read() core_data = re.sub(r'^(\S+:)$', r'__core_\1', core_data, flags=re.MULTILINE) settings = {k: v - for k, v in yaml.safe_load(core_data + '\n' + plugin_yaml).viewitems() + for k, v in yaml.safe_load(core_data + '\n' + plugin_yaml).items() if not k.startswith('__core_')} return {name: {'stylesheet': theme['stylesheet'], 'print_stylesheet': theme.get('print_stylesheet')} - for name, theme in settings.get('definitions', {}).viewitems() + for name, theme in settings.get('definitions', {}).items() if set(theme) & {'stylesheet', 'print_stylesheet'}} @@ -130,7 +130,7 @@ def _get_plugin_webpack_build_config(plugin_dir, url_root='/'): def _get_webpack_args(dev, watch): - args = ['--mode', 'development' if dev else 'production'] + args = ['--profile', '--progress', '--mode', 'development' if dev else 'production'] if watch: args.append('--watch') return args @@ -165,7 +165,7 @@ def _clean(webpack_build_config, plugin_dir=None): @cli.command('indico', short_help='Builds assets of Indico.') @_common_build_options() def build_indico(dev, clean, watch, url_root): - """Run webpack to build assets""" + """Run webpack to build assets.""" clean = clean or (clean is None and not dev) webpack_build_config_file = 'webpack-build-config.json' webpack_build_config = _get_webpack_build_config(url_root) @@ -188,10 +188,10 @@ def build_indico(dev, clean, watch, url_root): def _validate_plugin_dir(ctx, param, value): if not os.path.exists(os.path.join(value, 'setup.py')): - raise click.BadParameter('no setup.py found in {}'.format(value)) + raise click.BadParameter(f'no setup.py found in {value}') if (not os.path.exists(os.path.join(value, 'webpack.config.js')) and not os.path.exists(os.path.join(value, 'webpack-bundles.json'))): - raise click.BadParameter('no webpack.config.js or webpack-bundles.json found in {}'.format(value)) + raise click.BadParameter(f'no webpack.config.js or webpack-bundles.json found in {value}') return value @@ -219,7 +219,7 @@ def _chdir(path): callback=_validate_plugin_dir) @_common_build_options() def build_plugin(plugin_dir, dev, clean, watch, url_root): - """Run webpack to build plugin assets""" + """Run webpack to build plugin assets.""" clean = clean or (clean is None and not dev) webpack_build_config_file = os.path.join(plugin_dir, 'webpack-build-config.json') webpack_build_config = _get_plugin_webpack_build_config(plugin_dir, url_root) @@ -261,7 +261,7 @@ def build_plugin(plugin_dir, dev, clean, watch, url_root): @_common_build_options(allow_watch=False) @click.pass_context def build_all_plugins(ctx, plugins_dir, dev, clean, url_root): - """Run webpack to build plugin assets""" + """Run webpack to build plugin assets.""" plugins = sorted(d for d in os.listdir(plugins_dir) if _is_plugin_dir(os.path.join(plugins_dir, d))) for plugin in plugins: step('plugin: {}', plugin) diff --git a/bin/maintenance/build-wheel.py b/bin/maintenance/build-wheel.py index cdabe5b3ad1..a78cda74463 100755 --- a/bin/maintenance/build-wheel.py +++ b/bin/maintenance/build-wheel.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -51,7 +51,7 @@ def run(cmd, title, shell=False): try: subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=shell) except subprocess.CalledProcessError as exc: - fail('{} failed'.format(title), verbose_msg=exc.output) + fail(f'{title} failed', verbose_msg=exc.output) def build_assets(): @@ -112,7 +112,7 @@ def git_is_clean_indico(): ['git', 'diff', '--stat', '--color=always', '--staged'] + toplevel, ['git', 'clean', '-dn', '-e', '__pycache__'] + toplevel] for cmd in cmds: - rv = subprocess.check_output(cmd, stderr=subprocess.STDOUT) + rv = subprocess.check_output(cmd, stderr=subprocess.STDOUT, text=True) if rv: return False, rv return True, None @@ -123,8 +123,7 @@ def _iter_package_modules(package_masks): path = '/'.join(package.split('.')) if not os.path.exists(os.path.join(path, '__init__.py')): continue - for f in glob(os.path.join(path, '*.py')): - yield f + yield from glob(os.path.join(path, '*.py')) def _get_included_files(package_masks): @@ -141,7 +140,8 @@ def _get_included_files(package_masks): def _get_ignored_package_files_indico(): files = set(_get_included_files(('indico', 'indico.*'))) - output = subprocess.check_output(['git', 'ls-files', '--others', '--ignored', '--exclude-standard', 'indico/']) + output = subprocess.check_output(['git', 'ls-files', '--others', '--ignored', '--exclude-standard', 'indico/'], + text=True) ignored = {line for line in output.splitlines()} dist_path = 'indico/web/static/dist/' i18n_re = re.compile(r'^indico/translations/[a-zA-Z_]+/LC_MESSAGES/(?:messages\.mo|messages-react\.json)') @@ -164,14 +164,14 @@ def git_is_clean_plugin(): # plugins we don't have any package data to include anyway... cmds.append(['git', 'clean', '-dn', '-e', '__pycache__'] + toplevel) for cmd in cmds: - rv = subprocess.check_output(cmd, stderr=subprocess.STDOUT) + rv = subprocess.check_output(cmd, stderr=subprocess.STDOUT, text=True) if rv: return False, rv if not toplevel: # If we have just a single pyfile we don't need to check for ignored files return True, None rv = subprocess.check_output(['git', 'ls-files', '--others', '--ignored', '--exclude-standard'] + toplevel, - stderr=subprocess.STDOUT) + stderr=subprocess.STDOUT, text=True) garbage_re = re.compile(r'(\.(py[co]|mo)$)|(/__pycache__/)|(^({})/static/dist/)'.format('|'.join(toplevel))) garbage = [x for x in rv.splitlines() if not garbage_re.search(x)] if garbage: @@ -185,7 +185,7 @@ def patch_indico_version(add_version_suffix): def patch_plugin_version(add_version_suffix): - return _patch_version(add_version_suffix, 'setup.py', r"^(\s+)version='([^']+)'(,?)$", r"\1version='\2{}'\3") + return _patch_version(add_version_suffix, 'setup.cfg', r'^version = (.+)$', r'version = \1{}') @contextmanager @@ -203,10 +203,10 @@ def _patch_version(add_version_suffix, file_name, search, replace): if not add_version_suffix: yield return - rev = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).strip() + rev = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD'], text=True).strip() suffix = '+{}.{}'.format(datetime.now().strftime('%Y%m%d%H%M'), rev) info('adding version suffix: ' + suffix, unimportant=True) - with open(file_name, 'rb+') as f: + with open(file_name, 'r+') as f: old_content = f.read() f.seek(0) f.truncate() @@ -221,11 +221,13 @@ def _patch_version(add_version_suffix, file_name, search, replace): @click.group() -@click.option('--target-dir', '-d', type=click.Path(exists=True, file_okay=False, resolve_path=True), default='dist/', +@click.option('--target-dir', '-d', type=click.Path(file_okay=False, resolve_path=True), default='dist/', help='target dir for build wheels relative to the current dir') @click.pass_obj def cli(obj, target_dir): obj['target_dir'] = target_dir + if not os.path.exists(target_dir): + os.makedirs(target_dir) os.chdir(os.path.join(os.path.dirname(__file__), '..', '..')) @@ -235,7 +237,7 @@ def cli(obj, target_dir): @click.option('--ignore-unclean', is_flag=True, help='Ignore unclean working tree') @click.pass_obj def build_indico(obj, assets, add_version_suffix, ignore_unclean): - """Builds the indico wheel.""" + """Build the indico wheel.""" target_dir = obj['target_dir'] # check for unclean git status clean, output = git_is_clean_indico() @@ -261,7 +263,7 @@ def build_indico(obj, assets, add_version_suffix, ignore_unclean): def _validate_plugin_dir(ctx, param, value): if not os.path.exists(os.path.join(value, 'setup.py')): - raise click.BadParameter('no setup.py found in {}'.format(value)) + raise click.BadParameter(f'no setup.py found in {value}') return value @@ -278,7 +280,7 @@ def _plugin_has_assets(plugin_dir): @click.option('--ignore-unclean', is_flag=True, help='Ignore unclean working tree') @click.pass_obj def build_plugin(obj, assets, plugin_dir, add_version_suffix, ignore_unclean): - """Builds a plugin wheel. + """Build a plugin wheel. PLUGIN_DIR is the path to the folder containing the plugin's setup.py """ @@ -310,7 +312,7 @@ def build_plugin(obj, assets, plugin_dir, add_version_suffix, ignore_unclean): @click.option('--ignore-unclean', is_flag=True, help='Ignore unclean working tree') @click.pass_context def build_all_plugins(ctx, plugins_dir, add_version_suffix, ignore_unclean): - """Builds all plugin wheels in a directory. + """Build all plugin wheels in a directory. PLUGINS_DIR is the path to the folder containing the plugin directories """ diff --git a/bin/maintenance/dump_url_map.py b/bin/maintenance/dump_url_map.py index cf837fdf34e..0f9a7e9bc74 100644 --- a/bin/maintenance/dump_url_map.py +++ b/bin/maintenance/dump_url_map.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -18,7 +18,7 @@ def get_map_version(): # build a version identifier that is very likely to be different # whenever something changed h = hashlib.md5() - h.update(os.getcwd()) + h.update(os.getcwd().encode()) h.update(subprocess.check_output(['git', 'describe', '--always'])) h.update(subprocess.check_output(['git', 'status'])) h.update(subprocess.check_output(['git', 'diff'])) @@ -27,9 +27,9 @@ def get_map_version(): def get_rules(plugins): from indico.web.flask.app import make_app - app = make_app(set_path=True, testing=True, config_override={'BASE_URL': 'http://localhost/', - 'SECRET_KEY': '*' * 16, - 'PLUGINS': plugins}) + app = make_app(testing=True, config_override={'BASE_URL': 'http://localhost/', + 'SECRET_KEY': '*' * 16, + 'PLUGINS': plugins}) return dump_url_map(app.url_map) @@ -39,12 +39,13 @@ def get_rules(plugins): @click.option('-p', '--plugin', 'plugins', multiple=True, help='Include URLs from the specified plugins') def main(force, output, plugins): """ - Dumps the URL routing map to JSON for use by the `babel-flask-url` babel plugin. + Dump the URL routing map to JSON for use by the `babel-flask-url` + babel plugin. """ try: with open(output) as f: data = json.load(f) - except IOError: + except OSError: data = {} version = get_map_version() if not force and data.get('version') == version: diff --git a/bin/maintenance/make-release.py b/bin/maintenance/make-release.py index 48fd1a0f5de..512ad753d3f 100755 --- a/bin/maintenance/make-release.py +++ b/bin/maintenance/make-release.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -43,15 +43,14 @@ def run(cmd, title, shell=False): try: subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=shell) except subprocess.CalledProcessError as exc: - fail('{} failed'.format(title), verbose_msg=exc.output) + fail(f'{title} failed', verbose_msg=exc.output) def _bump_version(version): try: - parts = map(int, version.split('.')) + parts = [int(v) for v in version.split('.')] except ValueError: - fail('cannot bump version with non-numeric parts; did you forget --no-bump?') - sys.exit(1) + fail('cannot bump version with non-numeric parts') if len(parts) == 2: parts.append(0) parts[-1] += 1 @@ -59,7 +58,7 @@ def _bump_version(version): def _get_current_version(): - with open('indico/__init__.py', 'rb') as f: + with open('indico/__init__.py') as f: content = f.read() match = re.search(r"^__version__ = '([^']+)'$", content, re.MULTILINE) return match.group(1) @@ -67,31 +66,31 @@ def _get_current_version(): def _set_version(version, dry_run=False): step('Setting version to {}', version, dry_run=dry_run) - with open('indico/__init__.py', 'rb') as f: + with open('indico/__init__.py') as f: orig = content = f.read() - content = re.sub(r"^__version__ = '([^']+)'$", "__version__ = '{}'".format(version), content, flags=re.MULTILINE) + content = re.sub(r"^__version__ = '([^']+)'$", f"__version__ = '{version}'", content, flags=re.MULTILINE) assert content != orig if not dry_run: - with open('indico/__init__.py', 'wb') as f: + with open('indico/__init__.py', 'w') as f: f.write(content) def _set_changelog_date(new_version, dry_run=False): - with open('CHANGES.rst', 'rb') as f: + with open('CHANGES.rst') as f: orig = content = f.read() - version_line = 'Version {}'.format(new_version) + version_line = f'Version {new_version}' underline = '-' * len(version_line) unreleased = re.escape('Unreleased') release_date = format_date(format='MMMM dd, YYYY', locale='en') content = re.sub(r'(?<={}\n{}\n\n\*){}(?=\*\n)'.format(re.escape(version_line), underline, unreleased), - 'Released on {}'.format(release_date), + f'Released on {release_date}', content, flags=re.DOTALL) step('Setting release date to {}', release_date, dry_run=dry_run) if content == orig: fail('Could not update changelog - is there an entry for {}?', new_version) if not dry_run: - with open('CHANGES.rst', 'wb') as f: + with open('CHANGES.rst', 'w') as f: f.write(content) @@ -178,14 +177,14 @@ def cli(version, dry_run, sign, no_assets, no_changelog): if not no_changelog: _set_changelog_date(new_version, dry_run=dry_run) _set_version(new_version, dry_run=dry_run) - release_msg = 'Release {}'.format(new_version) + release_msg = f'Release {new_version}' _git_commit(release_msg, ['CHANGES.rst', 'indico/__init__.py'], dry_run=dry_run) _git_tag(new_version, release_msg, sign=sign, dry_run=dry_run) prompt = 'Build release wheel before bumping version?' if next_version else 'Build release wheel now?' if click.confirm(click.style(prompt, fg='blue', bold=True), default=True): _build_wheel(no_assets, dry_run=dry_run) if next_version: - next_message = 'Bump version to {}'.format(next_version) + next_message = f'Bump version to {next_version}' _set_version(next_version, dry_run=dry_run) _git_commit(next_message, ['indico/__init__.py'], dry_run=dry_run) diff --git a/bin/maintenance/update_backrefs.py b/bin/maintenance/update_backrefs.py index 03c84e1f08e..5353019c5cc 100644 --- a/bin/maintenance/update_backrefs.py +++ b/bin/maintenance/update_backrefs.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import sys from collections import defaultdict from operator import itemgetter @@ -14,24 +12,20 @@ import click from sqlalchemy import inspect -from indico.core.db import db -from indico.core.db.sqlalchemy.util.models import import_all_models +from indico.core.db.sqlalchemy.util.models import get_all_models, import_all_models from indico.util.console import cformat -click.disable_unicode_literals_warning = True - - def _find_backrefs(): backrefs = defaultdict(list) - for cls in db.Model._decl_class_registry.itervalues(): + for cls in get_all_models(): if not hasattr(cls, '__table__'): continue mapper = inspect(cls) for rel in mapper.relationships: if rel.backref is None: continue - backref_name = rel.backref if isinstance(rel.backref, basestring) else rel.backref[0] + backref_name = rel.backref if isinstance(rel.backref, str) else rel.backref[0] if cls != rel.class_attribute.class_: # skip relationships defined on a parent class continue @@ -45,7 +39,7 @@ def _get_source_file(cls): def _write_backrefs(rels, new_source): for backref_name, target, target_rel_name in sorted(rels, key=itemgetter(0)): - new_source.append(' # - {} ({}.{})'.format(backref_name, target, target_rel_name)) + new_source.append(f' # - {backref_name} ({target}.{target_rel_name})') @click.command() @@ -54,9 +48,9 @@ def _write_backrefs(rels, new_source): def main(ci): import_all_models() has_missing = has_updates = False - for cls, rels in sorted(_find_backrefs().iteritems(), key=lambda x: x[0].__name__): + for cls, rels in sorted(_find_backrefs().items(), key=lambda x: x[0].__name__): path = _get_source_file(cls) - with open(path, 'r') as f: + with open(path) as f: source = [line.rstrip('\n') for line in f] new_source = [] in_class = in_backrefs = backrefs_written = False @@ -76,7 +70,7 @@ def main(ci): # end of the indented class block in_class = False else: - if line.startswith('class {}('.format(cls.__name__)): + if line.startswith(f'class {cls.__name__}('): in_class = True new_source.append(line) if in_backrefs and not backrefs_written: diff --git a/bin/maintenance/update_browsers.js b/bin/maintenance/update_browsers.js index 83a0a54059f..2bf8b539bf1 100644 --- a/bin/maintenance/update_browsers.js +++ b/bin/maintenance/update_browsers.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/bin/maintenance/update_header.py b/bin/maintenance/update_header.py index 32df5429254..429eedede75 100644 --- a/bin/maintenance/update_header.py +++ b/bin/maintenance/update_header.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import os import re import subprocess @@ -14,44 +12,11 @@ from datetime import date import click +import yaml from indico.util.console import cformat -click.disable_unicode_literals_warning = True - - -HEADERS = { - 'indico': """ - {comment_start} This file is part of Indico. -{comment_middle} Copyright (C) 2002 - {end_year} CERN -{comment_middle} -{comment_middle} Indico is free software; you can redistribute it and/or -{comment_middle} modify it under the terms of the MIT License; see the -{comment_middle} LICENSE file for more details. -{comment_end} -""", - 'plugins': """ - {comment_start} This file is part of the Indico plugins. -{comment_middle} Copyright (C) 2002 - {end_year} CERN -{comment_middle} -{comment_middle} The Indico plugins are free software; you can redistribute -{comment_middle} them and/or modify them under the terms of the MIT License; -{comment_middle} see the LICENSE file for more details. -{comment_end} -""", - 'plugins-cern': """ - {comment_start} This file is part of the CERN Indico plugins. -{comment_middle} Copyright (C) 2014 - {end_year} CERN -{comment_middle} -{comment_middle} The CERN Indico plugins are free software; you can redistribute -{comment_middle} them and/or modify them under the terms of the MIT License; see -{comment_middle} the LICENSE file for more details. -{comment_end} -""", -} - - # Dictionary listing the files for which to change the header. # The key is the extension of the file (without the dot) and the value is another # dictionary containing two keys: @@ -63,28 +28,28 @@ # header. (See the `HEADER` above) SUPPORTED_FILES = { 'py': { - 'regex': re.compile(br'((^#|[\r\n]#).*)*'), - 'format': {'comment_start': b'#', 'comment_middle': b'#', 'comment_end': b''}}, + 'regex': re.compile(r'((^#|[\r\n]#).*)*'), + 'format': {'comment_start': '#', 'comment_middle': '#', 'comment_end': ''}}, 'wsgi': { - 'regex': re.compile(br'((^#|[\r\n]#).*)*'), - 'format': {'comment_start': b'#', 'comment_middle': b'#', 'comment_end': b''}}, + 'regex': re.compile(r'((^#|[\r\n]#).*)*'), + 'format': {'comment_start': '#', 'comment_middle': '#', 'comment_end': ''}}, 'js': { - 'regex': re.compile(br'/\*(.|[\r\n])*?\*/|((^//|[\r\n]//).*)*'), - 'format': {'comment_start': b'//', 'comment_middle': b'//', 'comment_end': b''}}, + 'regex': re.compile(r'/\*(.|[\r\n])*?\*/|((^//|[\r\n]//).*)*'), + 'format': {'comment_start': '//', 'comment_middle': '//', 'comment_end': ''}}, 'jsx': { - 'regex': re.compile(br'/\*(.|[\r\n])*?\*/|((^//|[\r\n]//).*)*'), - 'format': {'comment_start': b'//', 'comment_middle': b'//', 'comment_end': b''}}, + 'regex': re.compile(r'/\*(.|[\r\n])*?\*/|((^//|[\r\n]//).*)*'), + 'format': {'comment_start': '//', 'comment_middle': '//', 'comment_end': ''}}, 'css': { - 'regex': re.compile(br'/\*(.|[\r\n])*?\*/'), - 'format': {'comment_start': b'/*', 'comment_middle': b' *', 'comment_end': b' */'}}, + 'regex': re.compile(r'/\*(.|[\r\n])*?\*/'), + 'format': {'comment_start': '/*', 'comment_middle': ' *', 'comment_end': ' */'}}, 'scss': { - 'regex': re.compile(br'/\*(.|[\r\n])*?\*/|((^//|[\r\n]//).*)*'), - 'format': {'comment_start': b'//', 'comment_middle': b'//', 'comment_end': b''}}, + 'regex': re.compile(r'/\*(.|[\r\n])*?\*/|((^//|[\r\n]//).*)*'), + 'format': {'comment_start': '//', 'comment_middle': '//', 'comment_end': ''}}, } # The substring which must be part of a comment block in order for the comment to be updated by the header. -SUBSTRING = b'This file is part of' +SUBSTRING = 'This file is part of' USAGE = """ @@ -92,41 +57,80 @@ By default, all the files tracked by git in the current repository are updated to the current year. -You need to specify which project it is (one of {projects}) to the correct -headers are used. - You can specify a year to update to as well as a file or directory. This will update all the supported files in the scope including those not tracked by git. If the directory does not contain any supported files (or if the file specified is not supported) nothing will be updated. -""".format(supported_files=', '.join(SUPPORTED_FILES), projects=', '.join(HEADERS)).strip() +""".format(supported_files=', '.join(SUPPORTED_FILES)).strip() -def gen_header(project, data, end_year): - data['end_year'] = end_year - return '\n'.join(line.rstrip() for line in HEADERS[project].format(**data).strip().splitlines()).encode('ascii') +def _walk_to_root(path): + """Yield directories starting from the given directory up to the root.""" + # Based on code from python-dotenv (BSD-licensed): + # https://github.com/theskumar/python-dotenv/blob/e13d957b/src/dotenv/main.py#L245 + if os.path.isfile(path): + path = os.path.dirname(path) -def _update_header(project, file_path, year, substring, regex, data, ci): + last_dir = None + current_dir = os.path.abspath(path) + while last_dir != current_dir: + yield current_dir + parent_dir = os.path.abspath(os.path.join(current_dir, os.path.pardir)) + last_dir, current_dir = current_dir, parent_dir + + +def _get_config(path, end_year): + config = {} + for dirname in _walk_to_root(path): + check_path = os.path.join(dirname, 'headers.yml') + if os.path.isfile(check_path): + with open(check_path) as f: + config.update((k, v) for k, v in yaml.safe_load(f.read()).items() if k not in config) + if config.pop('root', False): + break + + if 'start_year' not in config: + click.echo('no valid headers.yml files found: start_year missing') + sys.exit(1) + if 'name' not in config: + click.echo('no valid headers.yml files found: name missing') + sys.exit(1) + if 'header' not in config: + click.echo('no valid headers.yml files found: header missing') + sys.exit(1) + config['end_year'] = end_year + return config + + +def gen_header(data): + if data['start_year'] == data['end_year']: + data['dates'] = data['start_year'] + else: + data['dates'] = '{} - {}'.format(data['start_year'], data['end_year']) + return '\n'.join(line.rstrip() for line in data['header'].format(**data).strip().splitlines()) + + +def _update_header(file_path, config, substring, regex, data, ci): found = False - with open(file_path, 'rb') as file_read: + with open(file_path) as file_read: content = orig_content = file_read.read() if not content.strip(): return False shebang_line = None - if content.startswith(b'#!/'): + if content.startswith('#!/'): shebang_line, content = content.split('\n', 1) for match in regex.finditer(content): if substring in match.group(): found = True - content = content[:match.start()] + gen_header(project, data, year) + content[match.end():] + content = content[:match.start()] + gen_header(dict(data, **config)) + content[match.end():] if shebang_line: content = shebang_line + '\n' + content if content != orig_content: msg = 'Incorrect header in {}' if ci else cformat('%{green!}Updating header of %{blue!}{}') print(msg.format(os.path.relpath(file_path))) if not ci: - with open(file_path, 'wb') as file_write: + with open(file_path, 'w') as file_write: file_write.write(content) return True elif not found: @@ -135,13 +139,14 @@ def _update_header(project, file_path, year, substring, regex, data, ci): return True -def update_header(project, file_path, year, ci): +def update_header(file_path, year, ci): + config = _get_config(file_path, year) ext = file_path.rsplit('.', 1)[-1] if ext not in SUPPORTED_FILES or not os.path.isfile(file_path): return False if os.path.basename(file_path)[0] == '.': return False - return _update_header(project, file_path, year, SUBSTRING, SUPPORTED_FILES[ext]['regex'], + return _update_header(file_path, config, SUBSTRING, SUPPORTED_FILES[ext]['regex'], SUPPORTED_FILES[ext]['format'], ci) @@ -164,13 +169,8 @@ def blacklisted(root, path, _cache={}): @click.option('--year', '-y', type=click.IntRange(min=1000), default=date.today().year, metavar='YEAR', help='Indicate the target year') @click.option('--path', '-p', type=click.Path(exists=True), help='Restrict updates to a specific file or directory') -@click.argument('project') @click.pass_context -def main(ctx, ci, project, year, path): - if project not in HEADERS: - click.echo(ctx.get_help()) - sys.exit(1) - +def main(ctx, ci, year, path): error = False if path and os.path.isdir(path): if not ci: @@ -179,34 +179,34 @@ def main(ctx, ci, project, year, path): for root, _, filenames in os.walk(path): for filename in filenames: if not blacklisted(path, root): - if update_header(project, os.path.join(root, filename), year, ci): + if update_header(os.path.join(root, filename), year, ci): error = True elif path and os.path.isfile(path): if not ci: print(cformat("Updating headers to the year %{yellow!}{year}%{reset} for the file " "%{yellow!}{file}%{reset}...").format(year=year, file=path)) - if update_header(project, path, year, ci): + if update_header(path, year, ci): error = True else: if not ci: print(cformat("Updating headers to the year %{yellow!}{year}%{reset} for all " "git-tracked files...").format(year=year)) try: - for filepath in subprocess.check_output(['git', 'ls-files']).splitlines(): + for filepath in subprocess.check_output(['git', 'ls-files'], text=True).splitlines(): filepath = os.path.abspath(filepath) if not blacklisted(os.getcwd(), os.path.dirname(filepath)): - if update_header(project, filepath, year, ci): + if update_header(filepath, year, ci): error = True except subprocess.CalledProcessError: raise click.UsageError(cformat('%{red!}You must be within a git repository to run this script.')) if not error: - print(cformat('%{green}\u2705 All headers are up to date').encode('utf-8')) + print(cformat('%{green}\u2705 All headers are up to date')) elif ci: - print(cformat('%{red}\u274C Some headers need to be updated or added').encode('utf-8')) + print(cformat('%{red}\u274C Some headers need to be updated or added')) sys.exit(1) else: - print(cformat('%{yellow}\U0001F504 Some headers have been updated (or are missing)').encode('utf-8')) + print(cformat('%{yellow}\U0001F504 Some headers have been updated (or are missing)')) if __name__ == '__main__': diff --git a/bin/utils/apiProxy.py b/bin/utils/apiProxy.py index e3fa87f2ed8..ebf81b34c81 100644 --- a/bin/utils/apiProxy.py +++ b/bin/utils/apiProxy.py @@ -1,19 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function - import hashlib import hmac import optparse import sys import time -import urllib from contextlib import closing +from urllib.parse import urlencode import requests from flask import Flask, Response, abort, request @@ -24,7 +22,7 @@ def build_indico_request(path, params, api_key=None, secret_key=None, only_public=False): - items = params.items() if hasattr(params, 'items') else list(params) + items = list(params.items()) if hasattr(params, 'items') else list(params) if api_key: items.append(('apikey', api_key)) if only_public: @@ -32,8 +30,8 @@ def build_indico_request(path, params, api_key=None, secret_key=None, only_publi if secret_key: items.append(('timestamp', str(int(time.time())))) items = sorted(items, key=lambda x: x[0].lower()) - url = '%s?%s' % (path, urllib.urlencode(items)) - signature = hmac.new(secret_key, url, hashlib.sha1).hexdigest() + url = '{}?{}'.format(path, urlencode(items)) + signature = hmac.new(secret_key.encode(), url.encode(), hashlib.sha1).hexdigest() items.append(('signature', signature)) return items diff --git a/bin/utils/create_module.py b/bin/utils/create_module.py index 9be4cee94f0..ff74e82dac8 100644 --- a/bin/utils/create_module.py +++ b/bin/utils/create_module.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import re import textwrap @@ -14,7 +12,6 @@ from datetime import date import click -click.disable_unicode_literals_warning = True def _validate_indico_dir(ctx, param, value): @@ -63,12 +60,12 @@ def touch(path): def write(f, text=''): if text: - f.write(text.encode('ascii')) - f.write(b'\n') + f.write(text) + f.write('\n') def write_header(f): - f.write(textwrap.dedent(b""" + f.write(textwrap.dedent(""" # This file is part of Indico. # Copyright (C) 2002 - {year} CERN # @@ -76,8 +73,6 @@ def write_header(f): # modify it under the terms of the MIT License; see the # LICENSE file for more details. - from __future__ import unicode_literals - """).lstrip().format(year=date.today().year)) @@ -86,8 +81,8 @@ def write_model(f, class_name, event): table_name = _snakify(class_name) + 's' if event and table_name.startswith('event_'): table_name = table_name[6:] - f.write(b'\n\n') - f.write(textwrap.dedent(b""" + f.write('\n\n') + f.write(textwrap.dedent(""" class {cls}(db.Model): __tablename__ = '{table}' __table_args__ = {{'schema': '{schema}'}} @@ -98,7 +93,6 @@ class {cls}(db.Model): primary_key=True ) - @return_ascii def __repr__(self): return format_repr(self, 'id') """).lstrip().format(cls=class_name, table=table_name, schema=schema)) @@ -128,18 +122,18 @@ def main(indico_dir, name, module_dir, event, models, blueprint, templates, cont if not os.path.exists(models_dir): os.mkdir(models_dir) touch(os.path.join(models_dir, '__init__.py')) - for module_name, class_names in model_classes.iteritems(): - model_path = os.path.join(models_dir, '{}.py'.format(module_name)) + for module_name, class_names in model_classes.items(): + model_path = os.path.join(models_dir, f'{module_name}.py') if os.path.exists(model_path): - raise click.exceptions.UsageError('Cannot create model in {} (file already exists)'.format(module_name)) + raise click.exceptions.UsageError(f'Cannot create model in {module_name} (file already exists)') with open(model_path, 'w') as f: write_header(f) write(f, 'from indico.core.db import db') - write(f, 'from indico.util.string import format_repr, return_ascii') + write(f, 'from indico.util.string import format_repr') for class_name in class_names: write_model(f, class_name, event) if blueprint: - blueprint_name = 'event_{}'.format(name) if event else name + blueprint_name = f'event_{name}' if event else name blueprint_path = os.path.join(module_dir, 'blueprint.py') if os.path.exists(blueprint_path): raise click.exceptions.UsageError('Cannot create blueprint (file already exists)') @@ -148,11 +142,11 @@ def main(indico_dir, name, module_dir, event, models, blueprint, templates, cont write(f, 'from indico.web.flask.wrappers import IndicoBlueprint') write(f) if templates: - virtual_template_folder = 'events/{}'.format(name) if event else name + virtual_template_folder = f'events/{name}' if event else name write(f, "_bp = IndicoBlueprint('{}', __name__, template_folder='templates',\n\ virtual_template_folder='{}')".format(blueprint_name, virtual_template_folder)) else: - write(f, "_bp = IndicoBlueprint('{}', __name__)".format(blueprint_name)) + write(f, f"_bp = IndicoBlueprint('{blueprint_name}', __name__)") write(f) if templates: templates_dir = os.path.join(module_dir, 'templates') diff --git a/bin/utils/db_diff.py b/bin/utils/db_diff.py index 57ef6c5cdea..d12a0809dab 100644 --- a/bin/utils/db_diff.py +++ b/bin/utils/db_diff.py @@ -1,14 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os -import pipes +import shlex import subprocess import sys @@ -18,46 +16,25 @@ from sqlalchemy import create_engine -click.disable_unicode_literals_warning = True - - def _indent(msg, level=4): indentation = level * ' ' return indentation + msg.replace('\n', '\n' + indentation) -def _subprocess_check_output(*popenargs, **kwargs): - # copied from subprocess.check_output; added the ability to pass data ta stdin - if 'stdout' in kwargs: - raise ValueError('stdout argument not allowed, it will be overridden.') - stdin_data = kwargs.pop('stdin_data', None) - if stdin_data: - kwargs['stdin'] = subprocess.PIPE - process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs) - output, unused_err = process.communicate(stdin_data) - retcode = process.poll() - if retcode: - cmd = kwargs.get('args') - if cmd is None: - cmd = popenargs[0] - raise subprocess.CalledProcessError(retcode, cmd, output=output) - return output - - -def _checked_call(verbose, args, return_output=False, env=None, stdin_data=None): - cmd = ' '.join([os.path.basename(args[0])] + map(pipes.quote, args[1:])) +def _checked_call(verbose, args, return_output=False, env=None, input=''): + cmd = ' '.join([os.path.basename(args[0]), shlex.join(args[1:])]) if verbose: - click.echo(click.style('** {}'.format(cmd), fg='blue', bold=True), err=True) + click.echo(click.style(f'** {cmd}', fg='blue', bold=True), err=True) kwargs = {} if env: kwargs['env'] = dict(os.environ, **env) try: - rv = _subprocess_check_output(args, stderr=subprocess.STDOUT, stdin_data=stdin_data, **kwargs).strip() + rv = subprocess.check_output(args, stderr=subprocess.STDOUT, input=input, text=True, **kwargs).strip() except OSError as exc: - click.echo(click.style('!! {} failed: {}'.format(cmd, exc), fg='red', bold=True), err=True) + click.echo(click.style(f'!! {cmd} failed: {exc}', fg='red', bold=True), err=True) sys.exit(1) except subprocess.CalledProcessError as exc: - click.echo(click.style('!! {} failed:'.format(cmd), fg='red', bold=True), err=True) + click.echo(click.style(f'!! {cmd} failed:', fg='red', bold=True), err=True) click.echo(click.style(_indent(exc.output.strip()), fg='yellow', bold=True), err=True) sys.exit(1) else: @@ -106,7 +83,7 @@ def _is_exe(fpath): @click.option('-r', '--reverse', help='Reverse - return instead the SQL to go from target to base', is_flag=True) def main(dbname, verbose, reverse): """ - Compares the structure of the database against what's created from the + Compare the structure of the database against what's created from the models during `indico db prepare`. By default the current database is assumed to be named `indico`, but it @@ -173,7 +150,7 @@ def main(dbname, verbose, reverse): click.echo(diff) else: pretty_diff = _checked_call(verbose, ['pygmentize', '-l', 'sql', '-f', 'terminal256', - '-O', 'style=native,bg=dark'], return_output=True, stdin_data=diff) + '-O', 'style=native,bg=dark'], return_output=True, input=diff) click.echo(pretty_diff) diff --git a/bin/utils/db_log.py b/bin/utils/db_log.py index f15835cf8a6..a8666ab3098 100644 --- a/bin/utils/db_log.py +++ b/bin/utils/db_log.py @@ -1,23 +1,21 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - -import cPickle import fcntl import logging.handlers import os +import pickle import pprint import re import signal -import SocketServer import struct import termios import textwrap +from socketserver import StreamRequestHandler, ThreadingTCPServer from threading import Lock import click @@ -28,12 +26,11 @@ from pygments.lexers.sql import SqlLexer -click.disable_unicode_literals_warning = True ignored_line_re = re.compile(r'^(?:(?P\d+):)?(?P.+?)(?::(?P\d+))?$') output_lock = Lock() -class LogRecordStreamHandler(SocketServer.StreamRequestHandler): +class LogRecordStreamHandler(StreamRequestHandler): def handle(self): while True: chunk = self.connection.recv(4) @@ -43,7 +40,7 @@ def handle(self): chunk = self.connection.recv(size) while len(chunk) < size: chunk = chunk + self.connection.recv(size - len(chunk)) - obj = cPickle.loads(chunk) + obj = pickle.loads(chunk) self.handle_log(obj) def _check_ignored_sources(self, source): @@ -121,11 +118,11 @@ def handle_log(self, obj): print_linesep() -class LogRecordSocketReceiver(SocketServer.ThreadingTCPServer): +class LogRecordSocketReceiver(ThreadingTCPServer): allow_reuse_address = True def __init__(self, host, port, traceback_frames, ignore_selects, ignored_sources, ignored_request_paths): - SocketServer.ThreadingTCPServer.__init__(self, (host, port), LogRecordStreamHandler) + ThreadingTCPServer.__init__(self, (host, port), LogRecordStreamHandler) self.timeout = 1 self.traceback_frames = traceback_frames self.ignore_selects = ignore_selects @@ -134,7 +131,7 @@ def __init__(self, host, port, traceback_frames, ignore_selects, ignored_sources def _process_path(self, path): regex = path[1:] if path[0] == '~' else re.escape(path) - return re.compile('^{}$'.format(regex)) + return re.compile(f'^{regex}$') def terminal_size(): @@ -143,12 +140,12 @@ def terminal_size(): def print_linesep(double=False, color=None): - char = u'\N{BOX DRAWINGS DOUBLE HORIZONTAL}' if double else u'\N{BOX DRAWINGS LIGHT HORIZONTAL}' + char = '\N{BOX DRAWINGS DOUBLE HORIZONTAL}' if double else '\N{BOX DRAWINGS LIGHT HORIZONTAL}' sep = terminal_size()[0] * char if color is None: print(sep) else: - print(u'\x1b[38;5;{}m{}\x1b[0m'.format(color, sep)) + print(f'\x1b[38;5;{color}m{sep}\x1b[0m') def indent(msg, level=4): @@ -157,7 +154,7 @@ def indent(msg, level=4): def prettify_caption(caption): - return '\x1b[38;5;75;04m{}\x1b[0m'.format(caption) + return f'\x1b[38;5;75;04m{caption}\x1b[0m' def prettify_duration(duration, is_total=False): @@ -166,13 +163,13 @@ def prettify_duration(duration, is_total=False): else: thresholds = [(0.25, 196), (0.1, 202), (0.05, 226), (0.005, 46), (None, 123)] color = next(c for t, c in thresholds if t is None or duration >= t) - return '\x1b[38;5;{}m{:.06f}s\x1b[0m'.format(color, duration) + return f'\x1b[38;5;{color}m{duration:.06f}s\x1b[0m' def prettify_count(count): thresholds = [(50, 196), (30, 202), (20, 226), (10, 46), (None, 123)] color = next(c for t, c in thresholds if t is None or count >= t) - return '\x1b[38;5;{}m{}\x1b[0m'.format(color, count) + return f'\x1b[38;5;{color}m{count}\x1b[0m' def prettify_source(source, traceback_frames): @@ -218,7 +215,7 @@ def sigint(*unused): '/assets/js-vars/user.js). Prefix with ~ to use a regex match instead of an exact string match.') def main(port, traceback_frames, ignore_selects, ignored_sources, ignored_request_paths): signal.signal(signal.SIGINT, sigint) - print('Listening on 127.0.0.1:{}'.format(port)) + print(f'Listening on 127.0.0.1:{port}') server = LogRecordSocketReceiver('localhost', port, traceback_frames=traceback_frames, ignore_selects=ignore_selects, ignored_sources=ignored_sources, ignored_request_paths=ignored_request_paths) diff --git a/bin/utils/room_occupancy.py b/bin/utils/room_occupancy.py index 576e517368f..2555d1e813a 100644 --- a/bin/utils/room_occupancy.py +++ b/bin/utils/room_occupancy.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function - from datetime import date import click @@ -15,6 +13,7 @@ from indico.modules.rb.models.locations import Location from indico.modules.rb.models.rooms import Room from indico.modules.rb.statistics import calculate_rooms_occupancy +from indico.util.string import natural_sort_key from indico.web.flask.app import make_app @@ -23,10 +22,10 @@ def _main(location): past_month = yesterday - relativedelta(days=29) past_year = yesterday - relativedelta(years=1) - if not location: - rooms = Room.find_all() - else: - rooms = Room.find_all(Location.name.in_(location), _join=Location) + query = Room.query + if location: + query = query.join(Location).filter(Location.name.in_(location)) + rooms = sorted(query, key=lambda r: natural_sort_key(r.location_name + r.full_name)) print('Month\tYear\tPublic?\tRoom') for room in rooms: diff --git a/bin/utils/storage_checksums.py b/bin/utils/storage_checksums.py index 6f63e23f68f..bc47e378a97 100644 --- a/bin/utils/storage_checksums.py +++ b/bin/utils/storage_checksums.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -48,12 +48,12 @@ def query_chunked(model, chunk_size): def main(): models = {model: make_query(model).count() for model in StoredFileMixin.__subclasses__()} - models = {model: total for model, total in models.iteritems() if total} + models = {model: total for model, total in models.items() if total} labels = {model: cformat('Processing %{blue!}{}%{reset} (%{cyan}{}%{reset} rows)').format(model.__name__, total) - for model, total in models.iteritems()} - max_length = max(len(x) for x in labels.itervalues()) - labels = {model: label.ljust(max_length) for model, label in labels.iteritems()} - for model, total in sorted(models.items(), key=itemgetter(1)): + for model, total in models.items()} + max_length = max(len(x) for x in labels.values()) + labels = {model: label.ljust(max_length) for model, label in labels.items()} + for model, total in sorted(list(models.items()), key=itemgetter(1)): with click.progressbar(query_chunked(model, 100), length=total, label=labels[model], show_percent=True, show_pos=True) as objects: for obj in objects: diff --git a/conftest.py b/conftest.py index 503d5ccc44c..6fb11bb6957 100644 --- a/conftest.py +++ b/conftest.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/docs/source/api/oauth.rst b/docs/source/api/oauth.rst index b19ebf5417d..91262432e3b 100644 --- a/docs/source/api/oauth.rst +++ b/docs/source/api/oauth.rst @@ -4,24 +4,16 @@ OAuth .. todo:: Docstrings (module, models, provider) -.. automodule:: indico.modules.oauth +.. automodule:: indico.core.oauth Models ++++++ -.. automodule:: indico.modules.oauth.models.applications +.. automodule:: indico.core.oauth.models.applications :members: :undoc-members: -.. automodule:: indico.modules.oauth.models.tokens - :members: - :undoc-members: - - -Utilities -+++++++++ - -.. automodule:: indico.modules.oauth.provider +.. automodule:: indico.core.oauth.models.tokens :members: :undoc-members: diff --git a/docs/source/building/index.rst b/docs/source/building/index.rst new file mode 100644 index 00000000000..58cb6c01c98 --- /dev/null +++ b/docs/source/building/index.rst @@ -0,0 +1,64 @@ +.. _building: + +Building +======== + +Before starting Indico compilation, this guide assumes you've previously +:ref:`setup the development base ` up until the :ref:`configuring step `. + +.. warning:: + We do not recommend doing these steps on the same system where you are running your production + version, as you run into the risk of mixing the latter with development resources. + +.. note:: + The ``master`` branch on Git is usually the next version under (heavy) development. Check if there + is a ``2.x-maintenance`` branch for your version and if yes, use that branch instead of ``master``. + +The first step is to generate a local distribution archive. Navigate to the Indico source folder +(by default, it is ``~/dev/indico/src``) and run the following command:: + + ./bin/maintenance/build-wheel.py indico --add-version-suffix + + +.. note:: + The build script refuses to run on a dirty git working directory, so any changes you decide to + include must be committed temporarily. You can use ``git checkout --detach`` to avoid committing + to your local master branch; if you plan to actually use the translation the better option would + be of course to create a real Git branch. + +.. warning:: + Make sure you're also not running any other build tool such as ``build-assets.py``, as it + may interfere with the creation of a production build when running in ``--watch`` mode. + +Finally, the ``dist`` folder will contain the wheel distribution, the file you should to copy to your production +machine:: + + dist/ + indico-2.3.1.dev0+202009231923.a14a24f564-py2-none-any.whl + + +To deploy this distribution, you should follow the :ref:`production installation guide `, +but instead of installing Indico from PyPI (``pip install indico``), install your custom-built wheel from +the previous step:: + + pip install /tmp/indico-2.3.1.dev0+202009231923.a14a24f564-py2-none-any.whl + +If you already have Indico installed, then simply installing the version from the wheel and restarting +uwsgi and indico-celery is all you need to do. + +Including a new translation +--------------------------- + +If you are including a new translation, you should also include the moment-js locale in +``indico/web/client/js/jquery/index.js`` before building:: + + // moment.js locales + import 'moment/locale/your-locale'; + + import 'moment/locale/zh-cn'; + import 'moment/locale/es'; + import 'moment/locale/fr'; + import 'moment/locale/en-gb'; + +.. note:: + Put your custom locale first, since ``en-gb`` needs to be the last one as a fallback. diff --git a/docs/source/conf.py b/docs/source/conf.py index 07df57fc641..07cd799a329 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -1,6 +1,4 @@ -# -*- coding: utf-8 -*- -# -# cds-indico documentation build configuration file, created by +# indico documentation build configuration file, created by # sphinx-quickstart on Sun Nov 29 13:19:24 2009. # # This file is execfile()d with the current directory set to its containing dir. @@ -30,8 +28,8 @@ def _get_version(): _version_re = re.compile(r'__version__\s+=\s+(.*)') - with open('../../indico/__init__.py', 'rb') as f: - version = str(ast.literal_eval(_version_re.search(f.read().decode('utf-8')).group(1))) + with open('../../indico/__init__.py') as f: + version = str(ast.literal_eval(_version_re.search(f.read()).group(1))) short_version = re.match(r'^(\d+(?:\.\d+)*).*?$', version).group(1) return short_version, version @@ -65,8 +63,8 @@ def _get_version(): master_doc = 'index' # General information about the project. -project = u'Indico' -copyright = u'2019, Indico Team' +project = 'Indico' +copyright = '2020, Indico Team' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -168,7 +166,7 @@ def _get_version(): # Custom sidebar templates, maps document names to template names. html_sidebars = { - '**': ['about.html', 'navigation.html', 'searchbox.html'] + '**': ['about.html', 'navigation.html', 'searchbox.html'] } # Additional templates that should be rendered to pages, maps page names to # template names. @@ -209,8 +207,7 @@ def _get_version(): # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, documentclass [howto/manual]). latex_documents = [ - ('index', 'indico.tex', u'Indico Documentation', - u'Indico Team', 'manual'), + ('index', 'indico.tex', 'Indico Documentation', 'Indico Team', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of diff --git a/docs/source/config/settings.rst b/docs/source/config/settings.rst index 9760eac17ca..946f88cc719 100644 --- a/docs/source/config/settings.rst +++ b/docs/source/config/settings.rst @@ -65,6 +65,26 @@ Authentication Default: ``False`` +.. data:: FAILED_LOGIN_RATE_LIMIT + + Applies a rate limit to failed login attempts due to an invalid username + or password. When specifying multiple rate limits separated with a semicolon, + they are checked in that specific order, which can allow for a short burst of + attempts (e.g. a legitimate user trying multiple passwords they commonly use) + and then slowing down more strongly (in case someone tries to brute-force more + than just a few passwords). + + Rate limiting is applied by IP address and only failed logins count against the + rate limit. It also does not apply to login attempts using external login systems + (SSO) as failures there are rarely related to invalid credentials coming from the + user (these would be rejected on the SSO side, which should implement its own rate + limiting). + + The default allows a burst of 15 attempts, and then only 5 attempts every 15 + minutes for the next 24 hours. Setting the rate limit to ``None`` disables it. + + Default: ``'5 per 15 minutes; 10 per day'`` + .. data:: EXTERNAL_REGISTRATION_URL The URL to an external page where people can register an account that @@ -104,33 +124,9 @@ Authentication Cache ----- -.. data:: CACHE_BACKEND - - The backend used for caching. Valid backends are ``redis``, - ``files``, and ``memcached``. - - To use the ``redis`` backend (recommended), you need to set - :data:`REDIS_CACHE_URL` to the URL of your Redis instance. - - With the ``files`` backend, cache data is stored in :data:`CACHE_DIR`, - which always needs to be set, even when using a different cache - backend since Indico needs to cache some data on disk. - - To use the ``memcached`` backend, you need to install the - ``python-memcached`` package from PyPI and set :data:`MEMCACHED_SERVERS` - to a list containing at least one memcached server. - - .. note:: - - We only test Indico with the ``redis`` cache backend. While - the other backends should work, we make no guarantees as - they are not actively being used or tested. - - Default: ``'files'`` - .. data:: REDIS_CACHE_URL - The URL of the redis server to use with the ``redis`` cache backend. + The URL of the redis server to use for caching. If the Redis server requires authentication, use a URL like this: ``redis://unused:password@127.0.0.1:6379/1`` @@ -260,7 +256,7 @@ Customization {% extends '~footer.html' %} {% block footer_logo %} - {%- set filename = 'cern_small_light.png' if dark else 'cern_small.png' -%} + {%- set filename = 'cern_small_light.png' if dark|default(false) else 'cern_small.png' -%} @@ -512,6 +508,20 @@ Emails Default: ``None`` +Experimental Features +---------------------- + +.. data:: EXPERIMENTAL_EDITING_SERVICE + + If enabled, event managers can connect the Editing module of their + events to an external microservice extending the normal Editing workflow. + As long as this is considered experimental, there are no guarantees + on backwards compatibility even in minor Indico version bumps. Please + check the `reference implementation`_ for details/changes. + + Default: ``False`` + + LaTeX ----- @@ -793,31 +803,11 @@ System Default: ``socket.getfqdn()`` -.. data:: FLOWER_URL - - The URL of the `Flower`_ instance monitoring your Celery workers. - If set, a link to it will be displayed in the admin area. - - To use flower, install it using ``pip install flower``, then start - it using ``indico celery flower``. By default it will listen on the - same host as specified in :data:`BASE_URL` (plain HTTP) on port 5555. - Authentication is done using OAuth so only Indico administrators - can access flower. You need to configure the allowed auth callback - URLs in the admin area; otherwise authentication will fail with an - OAuth error. - - .. note:: - - The information displayed by Flower is usually not very useful. - Unless you are very curious it is usually not worth using it. - - Default: ``None`` - .. _Flask-SQLAlchemy documentation: https://flask-sqlalchemy.readthedocs.io/en/stable/config/#configuration-keys .. _Sentry: https://sentry.io .. _Celery documentation on brokers: https://celery.readthedocs.io/en/stable/getting-started/brokers/index.html .. _Celery documentation on periodic tasks: https://celery.readthedocs.io/en/stable/userguide/periodic-tasks.html#available-fields -.. _Flower: https://flower.readthedocs.io/en/latest/ .. _TeXLive: https://www.tug.org/texlive/ .. _Flask-Multipass: https://flask-multipass.readthedocs.io +.. _reference implementation: https://github.com/indico/openreferee diff --git a/docs/source/exec_directive.py b/docs/source/exec_directive.py index 5d6cad34068..dc65ec6cfc3 100644 --- a/docs/source/exec_directive.py +++ b/docs/source/exec_directive.py @@ -2,7 +2,7 @@ import os import sys -from cStringIO import StringIO +from io import StringIO from docutils import nodes from docutils.parsers.rst import Directive @@ -10,7 +10,10 @@ class ExecDirective(Directive): - """Execute the specified python code and insert the output into the document""" + """ + Execute the specified python code and insert the output into the document. + """ + has_content = True def run(self): @@ -19,7 +22,7 @@ def run(self): old_stdout, sys.stdout = sys.stdout, StringIO() try: - exec '\n'.join(self.content) + exec('\n'.join(self.content)) text = sys.stdout.getvalue() lines = string2lines(text, tab_width, convert_whitespace=True) self.state_machine.insert_input(lines, source) diff --git a/docs/source/http_api/common.rst b/docs/source/http_api/common.rst index e1d3a3823af..7c57c44d240 100644 --- a/docs/source/http_api/common.rst +++ b/docs/source/http_api/common.rst @@ -19,6 +19,7 @@ onlyauthed oa Fail if the request is unauthenticated for any reason when this is set to *yes*. cookieauth ca Use the Indico session cookie to authenticate instead of an API key. +nocache nc Disable caching of results when this is set to *yes*. limit n Return no more than the X results. offset O Skip the first X results. detail d Specify the detail level (values depend on the exported diff --git a/docs/source/http_api/exporters/user.rst b/docs/source/http_api/exporters/user.rst index 69c3bf7ae22..8bf5886b7a5 100644 --- a/docs/source/http_api/exporters/user.rst +++ b/docs/source/http_api/exporters/user.rst @@ -22,33 +22,27 @@ None Results -------------- +------- Returns the user information (or an error in *JSON* format). -Result for https://indico.server/export/user/36024.json?ak=00000000-0000-0000-0000-000000000000&pretty=yes:: +Result for https://indico.server/export/user/6.json?ak=00000000-0000-0000-0000-000000000000&pretty=yes:: { "count": 1, "additionalInfo": {}, - "_type": "HTTPAPIResult", - "complete": true, - "url": "https:\/\/indico.server\/export\/user\/36024.json?ak=00000000-0000-0000-0000-000000000000&pretty=yes", - "ts": 1367243741, - "results": [ - { - "_type": "Avatar", - "name": "Alberto RESCO PEREZ", - "firstName": "Alberto", - "affiliation": "CERN", - "familyName": "Resco Perez", + "_type": "HTTPAPIResult" + "ts": 1610536660, + "url": "https:\/\/indico.server\/export\/user\/6.json?ak=00000000-0000-0000-0000-000000000000&pretty=yes", + "results": [{ + "id": 6, + "first_name": "Guinea", + "last_name": "Pig", + "full_name": "Guinea Pig" "email": "test@cern.ch", - "phone": "+41XXXXXXXXX", - "_fossil": "avatar", - "title": "", - "id": "36024" - } - ] + "affiliation": "CERN", + "phone": "", + "avatar_url": "\/user\/6\/picture-default", + "identifier": "User:6", + }], } - - diff --git a/docs/source/index.rst b/docs/source/index.rst index 84fcc117dae..83f1a98712a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -39,6 +39,15 @@ Configuration config/index.rst +Building +++++++++ + +.. toctree:: + :maxdepth: 2 + + building/index.rst + + Plugins +++++++ diff --git a/docs/source/indico_uml_directive.py b/docs/source/indico_uml_directive.py index b996c55d194..523b49d4bb8 100644 --- a/docs/source/indico_uml_directive.py +++ b/docs/source/indico_uml_directive.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2019 CERN +# Copyright (C) 2002 - 2020 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -26,7 +26,7 @@ def run(self): for n, l in enumerate(self.content, n + 1): content.append(l, source='diagram', offset=n) self.content = content - return super(IndicoUMLDirective, self).run() + return super().run() def _get_directive_name(self): return 'indico_uml' diff --git a/docs/source/installation/development.rst b/docs/source/installation/development.rst index 8cba9f68fe1..2d0984ad00a 100644 --- a/docs/source/installation/development.rst +++ b/docs/source/installation/development.rst @@ -10,15 +10,25 @@ Web assets such as JavaScript and SCSS files are compiled using `Webpack `_. -Do not use the default NodeJS packages from your Linux distribution as they are usually outdated or come wit +Do not use the default NodeJS packages from your Linux distribution as they are usually outdated or come with an outdated npm version. +Since only few Linux distributions include Python 3.9 in their package managers, we recommend installing +`pyenv `_ and then install the latest Python 3.9 version using +``pyenv install 3.9.1`` (adapt this command in case a newer version is available). + +.. tip:: + + You can run ``pyenv doctor`` once you installed and enabled pyenv in order to see whether all dependencies are + met. There's a good chance that you need to install some additional system packages beyond those listed below, and using + this tool will tell you what exactly you need. + CentOS/Fedora +++++++++++++ .. code-block:: shell - yum install -y gcc redis python-devel python-virtualenv libjpeg-turbo-devel libxslt-devel libxml2-devel \ + yum install -y gcc redis libjpeg-turbo-devel libxslt-devel libxml2-devel \ libffi-devel pcre-devel libyaml-devel redhat-rpm-config \ postgresql postgresql-server postgresql-contrib libpq-devel systemctl start redis.service postgresql.service @@ -29,7 +39,7 @@ Debian/Ubuntu .. code-block:: shell - apt install -y --install-recommends python-dev python-virtualenv libxslt1-dev libxml2-dev libffi-dev libpcre3-dev \ + apt install -y --install-recommends libxslt1-dev libxml2-dev libffi-dev libpcre3-dev \ libyaml-dev build-essential redis-server postgresql libpq-dev Then on Debian:: @@ -46,14 +56,8 @@ macOS We recommend that you use `Homebrew `_:: - brew install python2 redis libjpeg libffi pcre libyaml postgresql + brew install redis libjpeg libffi pcre libyaml postgresql brew services start postgresql - pip install virtualenv - -Note: Homebrew dropped support for the python2 formula at the end of 2019. -As an alternative you can install it directly using the latest commit:: - - brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/86a44a0a552c673a05f11018459c9f5faae3becc/Formula/python@2.rb Creating the directory structure @@ -69,8 +73,17 @@ developers keep all their code inside a ``dev`` or ``code`` dir. We will assume We will need a virtualenv where to run Indico:: cd ~/dev/indico - virtualenv env -p /usr/bin/python2.7 + pyenv local 3.9.1 + python -m venv env +.. note:: + + After setting the version with pyenv, it's a good idea to use ``python -V`` to ensure you are really running that + particular Python version; depending on the shell you may need to restart your shell first. In case you installed + a newer version than 3.9.1 earlier, adapt the pyenv command accordingly. + + +.. _cloning: Cloning Indico -------------- @@ -103,8 +116,8 @@ unnecessarily. This is why we advise that you include a fake SMTP server in your `Maildump `_ does exactly this and runs on Python. It should be quite simple to set up:: - virtualenv maildump -p /usr/bin/python2.7 - ./maildump/bin/pip install -U pip setuptools + python -m venv maildump + ./maildump/bin/pip install -U pip setuptools wheel ./maildump/bin/pip install maildump ./maildump/bin/maildump -p /tmp/maildump.pid @@ -122,17 +135,18 @@ Creating the DB createdb indico -T indico_template +.. _configuring-dev: + Configuring ----------- Let's get into the Indico virtualenv:: source ./env/bin/activate - pip install -U pip setuptools + pip install -U pip setuptools wheel cd src - pip install -r requirements.dev.txt - pip install -e . + pip install -e '.[dev]' npm ci Then, follow the instructions given by the wizard:: diff --git a/docs/source/installation/production/centos/apache.rst b/docs/source/installation/production/centos/apache.rst index 03d483361ed..ffd9c15b64e 100644 --- a/docs/source/installation/production/centos/apache.rst +++ b/docs/source/installation/production/centos/apache.rst @@ -26,7 +26,7 @@ to the ``[base]`` and ``[updates]`` sections, as described in the .. code-block:: shell - yum install -y yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm + yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm yum install -y postgresql96 postgresql96-server postgresql96-libs postgresql96-devel postgresql96-contrib yum install -y httpd mod_proxy_uwsgi mod_ssl mod_xsendfile yum install -y gcc redis uwsgi uwsgi-plugin-python2 @@ -136,6 +136,10 @@ most cases. LogLevel error ServerSignature Off + + Redirect 301 / https://YOURHOSTNAME/ + + AliasMatch "^/(images|fonts)(.*)/(.+?)(__v[0-9a-f]+)?\.([^.]+)$" "/opt/indico/web/static/$1$2/$3.$5" AliasMatch "^/(css|dist|images|fonts)/(.*)$" "/opt/indico/web/static/$1/$2" Alias /robots.txt /opt/indico/web/static/robots.txt diff --git a/docs/source/installation/production/centos/nginx.rst b/docs/source/installation/production/centos/nginx.rst index 959a6c60fdd..1dea7388ebe 100644 --- a/docs/source/installation/production/centos/nginx.rst +++ b/docs/source/installation/production/centos/nginx.rst @@ -28,7 +28,7 @@ to the ``[base]`` and ``[updates]`` sections, as described in the .. code-block:: shell - yum install -y yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm + yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm yum install -y postgresql96 postgresql96-server postgresql96-libs postgresql96-devel postgresql96-contrib yum install -y gcc redis nginx uwsgi uwsgi-plugin-python2 yum install -y python-devel python-virtualenv libjpeg-turbo-devel libxslt-devel libxml2-devel libffi-devel pcre-devel libyaml-devel @@ -136,6 +136,10 @@ most cases. access_log /opt/indico/log/nginx/access.log combined; error_log /opt/indico/log/nginx/error.log; + if ($host != $server_name) { + rewrite ^/(.*) https://$server_name/$1 permanent; + } + location /.xsf/indico/ { internal; alias /opt/indico/; diff --git a/docs/source/installation/production/debian/apache.rst b/docs/source/installation/production/debian/apache.rst index 899fd99b033..5ea5c374684 100644 --- a/docs/source/installation/production/debian/apache.rst +++ b/docs/source/installation/production/debian/apache.rst @@ -138,6 +138,10 @@ most cases. LogLevel error ServerSignature Off + + Redirect 301 / https://YOURHOSTNAME/ + + AliasMatch "^/(images|fonts)(.*)/(.+?)(__v[0-9a-f]+)?\.([^.]+)$" "/opt/indico/web/static/$1$2/$3.$5" AliasMatch "^/(css|dist|images|fonts)/(.*)$" "/opt/indico/web/static/$1/$2" Alias /robots.txt /opt/indico/web/static/robots.txt diff --git a/docs/source/installation/production/debian/nginx.rst b/docs/source/installation/production/debian/nginx.rst index d5535aae6b9..9f38e960ab4 100644 --- a/docs/source/installation/production/debian/nginx.rst +++ b/docs/source/installation/production/debian/nginx.rst @@ -144,6 +144,10 @@ most cases. access_log /opt/indico/log/nginx/access.log combined; error_log /opt/indico/log/nginx/error.log; + if ($host != $server_name) { + rewrite ^/(.*) https://$server_name/$1 permanent; + } + location /.xsf/indico/ { internal; alias /opt/indico/; diff --git a/docs/source/installation/translations.rst b/docs/source/installation/translations.rst index 59cd906c0da..0db8f69cef8 100644 --- a/docs/source/installation/translations.rst +++ b/docs/source/installation/translations.rst @@ -16,8 +16,11 @@ This is a guide to set up an Indico instance with a new language. It is useful for translators to verify how the translation looks in production or for administrators who just want to lurk at the incubated translation embryos. -Create your own Indico instance -------------------------------- +Alternatively, you may use this guide to expose a translation we do not officially support, +in your production version. + +1. Setup an Indico dev environment +---------------------------------- This should usually be done on your own computer or a virtual machine. @@ -27,16 +30,15 @@ it will prepare Indico to be served to users and used in all the different purpo The second is :ref:`development ` a light-weight, easier to set up, version oriented to testing purposes, that should not be exposed to the public. -For the purpose of translation **development** or **testing**, which this guide is about, -we recommend using the development version. +For the purpose of translation **development** or **testing** we recommend using the development version. -Install the transifex client ----------------------------- +2. Install the transifex client +------------------------------- Follow the instructions on the `transifex site `_. -Get an API token ----------------- +3. Get an API token +------------------- Go `to your transifex settings `_ and generate an API token. Afterwards, you should run the command ``tx init --skipsetup``. @@ -45,8 +47,8 @@ using transifex locally. If you do not know how to run this command, please refer to the `transifex client guide `_. -Install the translations ------------------------- +4. Install the translations +--------------------------- Navigate to ``~/dev/indico/src`` (assuming you used the standard locations from the dev setup guide). @@ -55,12 +57,17 @@ Languages codes can be obtained `here `_. For example, Chinese (China) is ``zh_CN.GB2312``. -Compile translations and run Indico ------------------------------------ +5. Compile translations and run Indico +-------------------------------------- Run the commands ``indico i18n compile-catalog`` and ``indico i18n compile-catalog-react`` -and :ref:`launch Indico `. +and: + +- :ref:`launch Indico `, or +- :ref:`build ` and :ref:`deploy your own version of Indico `, + if you wish to deploy the translation in a production version. + The language should now show up as an option in the top right corner. In case you modified the ``.js`` resources, you also need to delete the cached diff --git a/docs/source/plugins/models.rst b/docs/source/plugins/models.rst index 8c725af00ac..8c08cee0233 100644 --- a/docs/source/plugins/models.rst +++ b/docs/source/plugins/models.rst @@ -26,7 +26,6 @@ Plugins must describe its database model the in the *models* folder if needed:: backref=db.backref('example_foo', cascade='all, delete-orphan', lazy='dynamic'), ) - @return_ascii def __repr__(self): return u''.format(self.id, self.bar, self.location) diff --git a/headers.yml b/headers.yml new file mode 100644 index 00000000000..d83835ccd7e --- /dev/null +++ b/headers.yml @@ -0,0 +1,11 @@ +root: true +name: CERN +start_year: 2002 +header: |- + {comment_start} This file is part of Indico. + {comment_middle} Copyright (C) {dates} {name} + {comment_middle} + {comment_middle} Indico is free software; you can redistribute it and/or + {comment_middle} modify it under the terms of the MIT License; see the + {comment_middle} LICENSE file for more details. + {comment_end} diff --git a/indico/__init__.py b/indico/__init__.py index 4b2b3d1b501..9a38c0085c7 100644 --- a/indico/__init__.py +++ b/indico/__init__.py @@ -1,21 +1,13 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -import warnings - from indico.util.mimetypes import register_custom_mimetypes -__version__ = '2.3.1-dev' +__version__ = '3.0-dev' register_custom_mimetypes() - -# TODO: remove in 3.0 -warnings.filterwarnings('ignore', message='Python 2 is no longer supported by the Python core team.', - module='authlib') -warnings.filterwarnings('ignore', message='Python 2 is no longer supported by the Python core team.', - module='cryptography') diff --git a/indico/cli/cleanup.py b/indico/cli/cleanup.py index 923be788d2a..81044ad9949 100644 --- a/indico/cli/cleanup.py +++ b/indico/cli/cleanup.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import timedelta import click @@ -26,13 +24,13 @@ def _print_files(files): def cleanup_cmd(temp=False, cache=False, min_age=1, dry_run=False, verbose=False): if cache: if verbose: - click.echo(click.style('cleaning cache ({})'.format(config.CACHE_DIR), fg='white', bold=True)) + click.echo(click.style(f'cleaning cache ({config.CACHE_DIR})', fg='white', bold=True)) deleted = cleanup_dir(config.CACHE_DIR, timedelta(days=min_age), dry_run=dry_run) if verbose: _print_files(deleted) if temp: if verbose: - click.echo(click.style('cleaning temp ({})'.format(config.TEMP_DIR), fg='white', bold=True)) + click.echo(click.style(f'cleaning temp ({config.TEMP_DIR})', fg='white', bold=True)) deleted = cleanup_dir(config.TEMP_DIR, timedelta(days=min_age), dry_run=dry_run) if verbose: _print_files(deleted) diff --git a/indico/cli/core.py b/indico/cli/core.py index dee61332332..5e5f09fb43c 100644 --- a/indico/cli/core.py +++ b/indico/cli/core.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import click from flask.cli import AppGroup, pass_script_info @@ -17,7 +15,6 @@ from indico.cli.util import IndicoFlaskGroup, LazyGroup -click.disable_unicode_literals_warning = True __all__ = ('cli_command', 'cli_group') @@ -34,7 +31,7 @@ def _get_indico_version(ctx, param, value): if not value or ctx.resilient_parsing: return import indico - message = 'Indico v{}'.format(indico.__version__) + message = f'Indico v{indico.__version__}' click.echo(message, ctx.color) ctx.exit() diff --git a/indico/cli/database.py b/indico/cli/database.py index 8a993971fea..9f90d5f05eb 100644 --- a/indico/cli/database.py +++ b/indico/cli/database.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import os import sys from functools import partial @@ -45,20 +43,20 @@ def cli(ctx, plugin=None, all_plugins=False): @cli.command() def prepare(): - """Initializes a new database (creates tables, sets alembic rev to HEAD)""" + """Initialize a new database (creates tables, sets alembic rev to HEAD).""" return prepare_db() def _stamp(plugin=None, revision=None): - table = 'alembic_version' if not plugin else 'alembic_version_plugin_{}'.format(plugin) - db.session.execute('DELETE FROM {}'.format(table)) + table = 'alembic_version' if not plugin else f'alembic_version_plugin_{plugin}' + db.session.execute(f'DELETE FROM {table}') if revision: - db.session.execute('INSERT INTO {} VALUES (:revision)'.format(table), {'revision': revision}) + db.session.execute(f'INSERT INTO {table} VALUES (:revision)', {'revision': revision}) @cli.command() def reset_alembic(): - """Resets the alembic state carried over from 1.9.x + """Reset the alembic state carried over from 1.9.x. Only run this command right after upgrading from a 1.9.x version so the references to old alembic revisions (which were removed in @@ -88,16 +86,16 @@ def reset_alembic(): # All revisions were just data migrations -> get rid of them if plugin not in plugins: continue - print('[{}] Deleting revision table'.format(plugin)) - db.session.execute('DROP TABLE alembic_version_plugin_{}'.format(plugin)) + print(f'[{plugin}] Deleting revision table') + db.session.execute(f'DROP TABLE alembic_version_plugin_{plugin}') plugin_revisions = {'chat': '3888761f35f7', 'livesync': 'aa0dbc6c14aa', 'outlook': '6093a83228a7', 'vc_vidyo': '6019621fea50'} - for plugin, revision in plugin_revisions.iteritems(): + for plugin, revision in plugin_revisions.items(): if plugin not in plugins: continue - print('[{}] Stamping to new revision'.format(plugin)) + print(f'[{plugin}] Stamping to new revision') _stamp(plugin, revision) db.session.commit() @@ -115,8 +113,8 @@ def _safe_downgrade(*args, **kwargs): skip_confirm = False print(cformat('%{yellow!}***%{reset} ' "%{red!}Debug mode is NOT ACTIVE, so make sure you are on the right machine!")) - if not skip_confirm and raw_input(cformat('%{yellow!}***%{reset} ' - 'To confirm this, enter %{yellow!}YES%{reset}: ')) != 'YES': + if not skip_confirm and input(cformat('%{yellow!}***%{reset} ' + 'To confirm this, enter %{yellow!}YES%{reset}: ')) != 'YES': print(cformat('%{green}Aborted%{reset}')) sys.exit(1) else: @@ -131,7 +129,7 @@ def _call_with_plugins(*args, **kwargs): if plugin: plugins = {plugin_engine.get_plugin(plugin)} elif all_plugins: - plugins = set(plugin_engine.get_active_plugins().viewvalues()) + plugins = set(plugin_engine.get_active_plugins().values()) else: plugins = None @@ -150,7 +148,7 @@ def _call_with_plugins(*args, **kwargs): def _setup_cli(): - for command in flask_migrate_cli.commands.itervalues(): + for command in flask_migrate_cli.commands.values(): if command.name == 'init': continue command.callback = partial(with_appcontext(_call_with_plugins), _func=command.callback) diff --git a/indico/cli/devserver.py b/indico/cli/devserver.py index 2935d976c66..6124333b9a5 100644 --- a/indico/cli/devserver.py +++ b/indico/cli/devserver.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import os from flask.cli import DispatchingApp @@ -63,11 +61,11 @@ def run_server(info, host, port, url, ssl, ssl_key, ssl_cert, quiet, proxy, enab if not url: proto = 'https' if ssl else 'http' - url_host = '[{}]'.format(host) if ':' in host else host + url_host = f'[{host}]' if ':' in host else host if (port == 80 and not ssl) or (port == 443 and ssl): - url = '{}://{}'.format(proto, url_host) + url = f'{proto}://{url_host}' else: - url = '{}://{}:{}'.format(proto, url_host, port) + url = f'{proto}://{url_host}:{port}' os.environ['INDICO_DEV_SERVER'] = '1' os.environ.pop('FLASK_DEBUG', None) @@ -76,9 +74,9 @@ def run_server(info, host, port, url, ssl, ssl_key, ssl_cert, quiet, proxy, enab }) if os.environ.get('WERKZEUG_RUN_MAIN') != 'true': - print(' * Serving Indico on {}'.format(url)) + print(f' * Serving Indico on {url}') if evalex_whitelist: - print(' * Werkzeug debugger console on {}/console'.format(url)) + print(f' * Werkzeug debugger console on {url}/console') if evalex_whitelist is True: # noqa print(' * Werkzeug debugger console is available to all clients!') @@ -146,7 +144,7 @@ class DebuggedIndico(DebuggedApplication): def __init__(self, *args, **kwargs): self._evalex_whitelist = None self._request_ip = None - super(DebuggedIndico, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) @property def evalex(self): @@ -166,7 +164,7 @@ def __call__(self, environ, start_response): if self._request_ip.startswith('::ffff:'): # convert ipv6-style ipv4 to the regular ipv4 notation self._request_ip = self._request_ip[7:] - return super(DebuggedIndico, self).__call__(environ, start_response) + return super().__call__(environ, start_response) class QuietWSGIRequestHandler(WSGIRequestHandler): @@ -175,8 +173,8 @@ class QuietWSGIRequestHandler(WSGIRequestHandler): def log_request(self, code='-', size='-'): if code not in (304, 200): - super(QuietWSGIRequestHandler, self).log_request(code, size) + super().log_request(code, size) elif '?__debugger__=yes&cmd=resource' in self.path: pass # don't log debugger resources, they are quite uninteresting elif not any(self.path.startswith(self.INDICO_URL_PREFIX + x) for x in self.IGNORED_PATH_PREFIXES): - super(QuietWSGIRequestHandler, self).log_request(code, size) + super().log_request(code, size) diff --git a/indico/cli/event.py b/indico/cli/event.py index afb65c4e581..eb6833ae1db 100644 --- a/indico/cli/event.py +++ b/indico/cli/event.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import sys import click @@ -18,9 +16,6 @@ from indico.modules.users.models.users import User -click.disable_unicode_literals_warning = True - - @cli_group() def cli(): pass @@ -32,7 +27,7 @@ def cli(): help="The user which will be shown on the log as having restored the event (default: no user).") @click.option('-m', '--message', 'message', metavar="MESSAGE", help="An additional message for the log") def restore(event_id, user_id, message): - """Restores a deleted event.""" + """Restore a deleted event.""" event = Event.get(event_id) user = User.get(user_id) if user_id else None if event is None: @@ -42,17 +37,17 @@ def restore(event_id, user_id, message): click.secho('This event is not deleted', fg='yellow') sys.exit(1) event.is_deleted = False - text = 'Event restored: {}'.format(message) if message else 'Event restored' + text = f'Event restored: {message}' if message else 'Event restored' event.log(EventLogRealm.event, EventLogKind.positive, 'Event', text, user=user) db.session.commit() - click.secho('Event undeleted: "{}"'.format(event.title), fg='green') + click.secho(f'Event undeleted: "{event.title}"', fg='green') @cli.command() @click.argument('event_id', type=int) @click.argument('target_file', type=click.File('wb')) def export(event_id, target_file): - """Exports all data associated with an event. + """Export all data associated with an event. This exports the whole event as an archive which can be imported on another other Indico instance. Importing an event is only @@ -79,7 +74,7 @@ def export(event_id, target_file): @click.option('-c', '--category', 'category_id', type=int, default=0, metavar='ID', help='ID of the target category. Defaults to the root category.') def import_(source_file, create_users, force, verbose, yes, category_id): - """Imports an event exported from another Indico instance.""" + """Import an event exported from another Indico instance.""" click.echo('Importing event...') event = import_event(source_file, category_id, create_users=create_users, verbose=verbose, force=force) if event is None: diff --git a/indico/cli/i18n.py b/indico/cli/i18n.py index 1768022efb8..6256ba634d4 100644 --- a/indico/cli/i18n.py +++ b/indico/cli/i18n.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import json import os import re @@ -180,7 +178,7 @@ def check_format_strings(): invalid = _get_invalid_po_format_strings(path) if invalid: all_valid = False - click.echo('Found invalid format strings in {}'.format(os.path.relpath(path, root_path))) + click.echo(f'Found invalid format strings in {os.path.relpath(path, root_path)}') for item in invalid: click.echo(cformat('%{yellow}{}%{reset} | %{yellow!}{}%{reset}\n%{red}{}%{reset} != %{red!}{}%{reset}') .format(item['orig'], item['trans'], diff --git a/indico/cli/maintenance.py b/indico/cli/maintenance.py index e80919f984a..982abd8e0d4 100644 --- a/indico/cli/maintenance.py +++ b/indico/cli/maintenance.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import click from indico.cli.core import cli_group @@ -23,9 +21,6 @@ from indico.modules.events.sessions.models.principals import SessionPrincipal -click.disable_unicode_literals_warning = True - - @cli_group() def cli(): pass @@ -34,7 +29,7 @@ def cli(): def _fix_role_principals(principals, get_event): role_attrs = get_simple_column_attrs(EventRole) | {'members'} for p in principals: - click.echo('Fixing {}'.format(p)) + click.echo(f'Fixing {p}') event = get_event(p) try: event_role = [r for r in event.roles if r.code == p.event_role.code][0] @@ -42,14 +37,14 @@ def _fix_role_principals(principals, get_event): event_role = EventRole(event=event) event_role.populate_from_attrs(p.event_role, role_attrs) else: - click.echo(' using existing role {}'.format(event_role)) + click.echo(f' using existing role {event_role}') p.event_role = event_role db.session.flush() @cli.command() def fix_event_role_acls(): - """Fixes ACLs referencing event roles from other events. + """Fix ACLs referencing event roles from other events. This happened due to a bug prior to 2.2.3 when cloning an event which had event roles in its ACL. diff --git a/indico/cli/setup.py b/indico/cli/setup.py index 677c7b416c2..c3658e7e57e 100644 --- a/indico/cli/setup.py +++ b/indico/cli/setup.py @@ -1,18 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import re import shutil import socket import sys from operator import attrgetter +from pathlib import Path from smtplib import SMTP import click @@ -20,10 +19,8 @@ from flask.helpers import get_root_path from pkg_resources import iter_entry_points from prompt_toolkit import prompt -from prompt_toolkit.contrib.completers import PathCompleter, WordCompleter -from prompt_toolkit.layout.lexers import SimpleLexer -from prompt_toolkit.styles import style_from_dict -from prompt_toolkit.token import Token +from prompt_toolkit.completion import PathCompleter, WordCompleter +from prompt_toolkit.styles import Style from pytz import all_timezones, common_timezones from redis import RedisError, StrictRedis from sqlalchemy import create_engine @@ -37,9 +34,6 @@ from indico.util.string import validate_email -click.disable_unicode_literals_warning = True - - def _echo(msg=''): click.echo(msg, err=True) @@ -81,25 +75,25 @@ def _get_dirs(target_dir): return get_root_path('indico'), os.path.abspath(target_dir) -PROMPT_TOOLKIT_STYLE = style_from_dict({ - Token.HELP: '#aaaaaa', - Token.PROMPT: '#5f87ff', - Token.DEFAULT: '#dfafff', - Token.BRACKET: '#ffffff', - Token.COLON: '#ffffff', - Token.INPUT: '#aaffaa', +PROMPT_TOOLKIT_STYLE = Style.from_dict({ + 'help': '#aaaaaa', + 'prompt': '#5f87ff', + 'default': '#dfafff', + 'bracket': '#ffffff', + 'colon': '#ffffff', + '': '#aaffaa', # user input }) def _prompt(message, default='', path=False, list_=None, required=True, validate=None, allow_invalid=False, password=False, help=None): - def _get_prompt_tokens(cli): + def _get_prompt_tokens(): rv = [ - (Token.PROMPT, message), - (Token.COLON, ': '), + ('class:prompt', message), + ('class:colon', ': '), ] if first and help: - rv.insert(0, (Token.HELP, wrap_text(help) + '\n')) + rv.insert(0, ('class:help', wrap_text(help) + '\n')) return rv completer = None @@ -114,8 +108,8 @@ def _get_prompt_tokens(cli): first = True while True: try: - rv = prompt(get_prompt_tokens=_get_prompt_tokens, default=default, is_password=password, - completer=completer, lexer=SimpleLexer(Token.INPUT), style=PROMPT_TOOLKIT_STYLE) + rv = prompt(_get_prompt_tokens(), default=default, is_password=password, + completer=completer, style=PROMPT_TOOLKIT_STYLE) except (EOFError, KeyboardInterrupt): sys.exit(1) # pasting a multiline string works even with multiline disabled :( @@ -136,22 +130,22 @@ def _get_prompt_tokens(cli): def _confirm(message, default=False, abort=False, help=None): - def _get_prompt_tokens(cli): + def _get_prompt_tokens(): rv = [ - (Token.PROMPT, message), - (Token.BRACKET, ' ['), - (Token.DEFAULT, 'Y/n' if default else 'y/N'), - (Token.BRACKET, ']'), - (Token.COLON, ': '), + ('class:prompt', message), + ('class:bracket', ' ['), + ('class:default', 'Y/n' if default else 'y/N'), + ('class:bracket', ']'), + ('class:colon', ': '), ] if first and help: - rv.insert(0, (Token.HELP, wrap_text(help) + '\n')) + rv.insert(0, ('class:help', wrap_text(help) + '\n')) return rv first = True while True: try: - rv = prompt(get_prompt_tokens=_get_prompt_tokens, lexer=SimpleLexer(Token.INPUT), + rv = prompt(_get_prompt_tokens(), completer=WordCompleter(['yes', 'no'], ignore_case=True, sentence=True), style=PROMPT_TOOLKIT_STYLE) except (EOFError, KeyboardInterrupt): @@ -175,12 +169,12 @@ def _get_prompt_tokens(cli): @click.group() def cli(): - """This script helps with the initial steps of installing Indico""" + """This script helps with the initial steps of installing Indico.""" @cli.command() def list_plugins(): - """Lists the available indico plugins.""" + """List the available Indico plugins.""" import_all_models() table_data = [['Name', 'Title']] for ep in sorted(iter_entry_points('indico.plugins'), key=attrgetter('name')): @@ -193,7 +187,7 @@ def list_plugins(): @cli.command() @click.argument('target_dir') def create_symlinks(target_dir): - """Creates useful symlinks to run Indico from a webserver. + """Create useful symlinks to run Indico from a webserver. This lets you use static paths for the WSGI file and the htdocs folder so you do not need to update your webserver config when @@ -207,7 +201,7 @@ def create_symlinks(target_dir): @cli.command() @click.argument('target_dir') def create_logging_config(target_dir): - """Creates the default logging config file for Indico. + """Create the default logging config file for Indico. If a file already exists it is left untouched. This command is usually only used when doing a fresh indico installation when @@ -220,11 +214,11 @@ def create_logging_config(target_dir): @cli.command() @click.option('--dev', is_flag=True) def wizard(dev): - """Runs a setup wizard to configure Indico from scratch.""" + """Run a setup wizard to configure Indico from scratch.""" SetupWizard().run(dev=dev) -class SetupWizard(object): +class SetupWizard: def __init__(self): self._missing_dirs = set() self.root_path = None @@ -273,7 +267,7 @@ def _check_root(self): sys.exit(1) def _check_venv(self): - if not hasattr(sys, 'real_prefix'): + if sys.prefix == sys.base_prefix: _warn('It looks like you are not using a virtualenv. This is unsupported and strongly discouraged.') _prompt_abort() @@ -286,7 +280,6 @@ def _prompt_root_path(self, dev=False): # we want to go up a level since the data dir should not be # created inside the source directory default_root = os.path.dirname(default_root) - default_root = default_root.decode(sys.getfilesystemencoding()) else: default_root = '/opt/indico' self.root_path = _prompt('Indico root path', default=default_root, path=True, @@ -300,7 +293,7 @@ def _prompt_root_path(self, dev=False): if not sys.prefix.startswith(self.root_path + '/'): _warn('It is recommended to have the virtualenv inside the root directory, e.g. {}/.venv' .format(self.root_path)) - _warn('The virtualenv is currently located in {}'.format(sys.prefix)) + _warn(f'The virtualenv is currently located in {sys.prefix}') _prompt_abort() self.data_root_path = os.path.join(self.root_path, 'data') if dev else self.root_path @@ -352,7 +345,7 @@ def _check_url(url): return False return True - default_url = ('http://127.0.0.1:8000' if dev else 'https://{}'.format(socket.getfqdn())) + default_url = ('http://127.0.0.1:8000' if dev else f'https://{socket.getfqdn()}') url = _prompt('Indico URL', validate=_check_url, allow_invalid=True, default=default_url, help='Indico needs to know the URL through which it is accessible. ' @@ -367,7 +360,7 @@ def _check_postgres(uri, silent=False): engine.connect().close() except OperationalError as exc: if not silent: - _warn('Invalid database URI: ' + unicode(exc.orig).strip()) + _warn('Invalid database URI: ' + str(exc.orig).strip()) return False else: return True @@ -389,7 +382,7 @@ def _check_redis(uri): try: client.ping() except RedisError as exc: - _warn('Invalid redis URI: ' + unicode(exc)) + _warn('Invalid redis URI: ' + str(exc)) return False else: return True @@ -454,7 +447,7 @@ def _get_default_smtp(custom_host=None): help=('Indico needs an SMTP server to send emails.' if first else None)) if smtp_host != default_smtp_host: default_smtp_port = _get_default_smtp(smtp_host)[1] - smtp_port = int(_prompt('SMTP port', default=unicode(default_smtp_port or 25))) + smtp_port = int(_prompt('SMTP port', default=str(default_smtp_port or 25))) smtp_user = _prompt('SMTP username', default=smtp_user, required=False, help=('If your SMTP server requires authentication, ' 'enter the username now.' if first else None)) @@ -469,7 +462,7 @@ def _get_default_smtp(custom_host=None): smtp.login(smtp_user, smtp_password) smtp.quit() except Exception as exc: - _warn('SMTP connection failed: ' + unicode(exc)) + _warn('SMTP connection failed: ' + str(exc)) if not click.confirm('Keep these settings anyway?'): default_smtp_host = smtp_host default_smtp_port = smtp_port @@ -483,7 +476,8 @@ def _get_default_smtp(custom_host=None): def _prompt_defaults(self): def _get_all_locales(): # get all directories in indico/translations - return os.walk(os.path.join(get_root_path('indico'), 'translations')).next()[1] + root = Path(get_root_path('indico')) / 'translations' + return [ent.name for ent in root.iterdir() if ent.is_dir()] def _get_system_timezone(): candidates = [] @@ -498,7 +492,7 @@ def _get_system_timezone(): # as e.g. https://stackoverflow.com/a/12523283/298479 suggests # since this is ambiguous and we rather have the user type their # timezone if we don't have a very likely match. - return next((unicode(tz) for tz in candidates if tz in common_timezones), '') + return next((str(tz) for tz in candidates if tz in common_timezones), '') self.default_locale = _prompt('Default locale', default='en_GB', list_=_get_all_locales(), help='Specify the default language/locale used by Indico.') @@ -541,44 +535,43 @@ def _setup(self, dev=False): create_config_link = False config_data = [ - b'# General settings', - b'SQLALCHEMY_DATABASE_URI = {!r}'.format(self.db_uri.encode('utf-8')), - b'SECRET_KEY = {!r}'.format(os.urandom(32)), - b'BASE_URL = {!r}'.format(self.indico_url.encode('utf-8')), - b'CELERY_BROKER = {!r}'.format(self.redis_uri_celery.encode('utf-8')), - b'REDIS_CACHE_URL = {!r}'.format(self.redis_uri_cache.encode('utf-8')), - b"CACHE_BACKEND = 'redis'", - b'DEFAULT_TIMEZONE = {!r}'.format(self.default_timezone.encode('utf-8')), - b'DEFAULT_LOCALE = {!r}'.format(self.default_locale.encode('utf-8')), - b'ENABLE_ROOMBOOKING = {!r}'.format(self.rb_active), - b'CACHE_DIR = {!r}'.format(os.path.join(self.data_root_path, 'cache').encode('utf-8')), - b'TEMP_DIR = {!r}'.format(os.path.join(self.data_root_path, 'tmp').encode('utf-8')), - b'LOG_DIR = {!r}'.format(os.path.join(self.data_root_path, 'log').encode('utf-8')), - b'STORAGE_BACKENDS = {!r}'.format({k.encode('utf-8'): v.encode('utf-8') - for k, v in storage_backends.iteritems()}), - b"ATTACHMENT_STORAGE = 'default'", - b'ROUTE_OLD_URLS = True' if self.old_archive_dir else None, - b'', - b'# Email settings', - b'SMTP_SERVER = {!r}'.format((self.smtp_host.encode('utf-8'), self.smtp_port)), - b'SMTP_USE_TLS = {!r}'.format(bool(self.smtp_user and self.smtp_password)), - b'SMTP_LOGIN = {!r}'.format(self.smtp_user.encode('utf-8')), - b'SMTP_PASSWORD = {!r}'.format(self.smtp_password.encode('utf-8')), - b'SUPPORT_EMAIL = {!r}'.format(self.admin_email.encode('utf-8')), - b'PUBLIC_SUPPORT_EMAIL = {!r}'.format(self.contact_email.encode('utf-8')), - b'NO_REPLY_EMAIL = {!r}'.format(self.noreply_email.encode('utf-8')) + '# General settings', + f'SQLALCHEMY_DATABASE_URI = {self.db_uri!r}', + f'SECRET_KEY = {os.urandom(32)!r}', + f'BASE_URL = {self.indico_url!r}', + f'CELERY_BROKER = {self.redis_uri_celery!r}', + f'REDIS_CACHE_URL = {self.redis_uri_cache!r}', + f'DEFAULT_TIMEZONE = {self.default_timezone!r}', + f'DEFAULT_LOCALE = {self.default_locale!r}', + f'ENABLE_ROOMBOOKING = {self.rb_active!r}', + 'CACHE_DIR = {!r}'.format(os.path.join(self.data_root_path, 'cache')), + 'TEMP_DIR = {!r}'.format(os.path.join(self.data_root_path, 'tmp')), + 'LOG_DIR = {!r}'.format(os.path.join(self.data_root_path, 'log')), + 'STORAGE_BACKENDS = {!r}'.format({k: v + for k, v in storage_backends.items()}), + "ATTACHMENT_STORAGE = 'default'", + 'ROUTE_OLD_URLS = True' if self.old_archive_dir else None, + '', + '# Email settings', + f'SMTP_SERVER = {(self.smtp_host, self.smtp_port)!r}', + f'SMTP_USE_TLS = {bool(self.smtp_user and self.smtp_password)!r}', + f'SMTP_LOGIN = {self.smtp_user!r}', + f'SMTP_PASSWORD = {self.smtp_password!r}', + f'SUPPORT_EMAIL = {self.admin_email!r}', + f'PUBLIC_SUPPORT_EMAIL = {self.contact_email!r}', + f'NO_REPLY_EMAIL = {self.noreply_email!r}' ] if dev: config_data += [ - b'', - b'# Development options', - b'DB_LOG = True', - b'DEBUG = True', - b'SMTP_USE_CELERY = False' + '', + '# Development options', + 'DB_LOG = True', + 'DEBUG = True', + 'SMTP_USE_CELERY = False' ] - config = b'\n'.join(x for x in config_data if x is not None) + config = '\n'.join(x for x in config_data if x is not None) if dev: if not os.path.exists(self.data_root_path): @@ -590,8 +583,8 @@ def _setup(self, dev=False): os.mkdir(path) _echo(cformat('%{magenta}Creating %{magenta!}{}%{reset}%{magenta}').format(self.config_path)) - with open(self.config_path, 'wb') as f: - f.write(config + b'\n') + with open(self.config_path, 'w') as f: + f.write(config + '\n') package_root = get_root_path('indico') _copy(os.path.normpath(os.path.join(package_root, 'logging.yaml.sample')), diff --git a/indico/cli/shell.py b/indico/cli/shell.py index d6dd93b8065..918c00f8fea 100644 --- a/indico/cli/shell.py +++ b/indico/cli/shell.py @@ -1,23 +1,21 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import datetime import itertools import os import re import sys +from contextlib import ExitStack from functools import partial from operator import attrgetter, itemgetter import click import sqlalchemy.orm -from contextlib2 import ExitStack from flask import current_app import indico @@ -25,11 +23,11 @@ from indico.core.celery import celery from indico.core.config import config from indico.core.db import db +from indico.core.db.sqlalchemy.util.models import get_all_models from indico.core.plugins import plugin_engine from indico.modules.events import Event from indico.util.console import cformat from indico.util.date_time import now_utc, server_to_utc -from indico.util.fossilize import clearCache from indico.web.flask.stats import request_stats_request_started @@ -44,6 +42,8 @@ def _add_to_context(namespace, info, element, name=None, doc=None, color='green' def _add_to_context_multi(namespace, info, elements, names=None, doc=None, color='green'): + if not elements: + return if not names: names = [x.__name__ for x in elements] for name, element in zip(names, elements): @@ -58,17 +58,17 @@ def _add_to_context_smart(namespace, info, objects, get_name=attrgetter('__name_ def _get_module(obj): segments = tuple(obj.__module__.split('.')) if segments[0].startswith('indico_'): # plugin - return 'plugin:{}'.format(segments[0]) + return f'plugin:{segments[0]}' elif segments[:2] == ('indico', 'modules'): - return 'module:{}'.format(segments[2]) + return f'module:{segments[2]}' elif segments[:2] == ('indico', 'core'): - return 'core:{}'.format(segments[2]) + return f'core:{segments[2]}' else: return '.'.join(segments[:-1] if len(segments) > 1 else segments) items = [(_get_module(obj), get_name(obj), obj) for obj in objects] for module, items in itertools.groupby(sorted(items, key=itemgetter(0, 1)), key=itemgetter(0)): - names, elements = zip(*((x[1], x[2]) for x in items)) + names, elements = list(zip(*((x[1], x[2]) for x in items))) _add_to_context_multi(namespace, info, elements, names, doc=module, color=color) @@ -89,18 +89,19 @@ def _make_shell_context(): color='yellow') # Models info.append(cformat('*** %{magenta!}Models%{reset} ***')) - models = [cls for name, cls in sorted(db.Model._decl_class_registry.items(), key=itemgetter(0)) - if hasattr(cls, '__table__')] + models = [cls for cls in sorted(get_all_models(), key=attrgetter('__name__')) if hasattr(cls, '__table__')] add_to_context_smart(models) # Tasks info.append(cformat('*** %{magenta!}Tasks%{reset} ***')) - tasks = [task for task in sorted(celery.tasks.values()) if not task.name.startswith('celery.')] + tasks = [task for task in sorted(celery.tasks.values(), key=attrgetter('name')) + if not task.name.startswith('celery.')] add_to_context_smart(tasks, get_name=lambda x: x.name.replace('.', '_'), color='blue!') # Plugins - info.append(cformat('*** %{magenta!}Plugins%{reset} ***')) - plugins = [type(plugin) for plugin in sorted(plugin_engine.get_active_plugins().values(), + plugins = [type(plugin) for plugin in sorted(list(plugin_engine.get_active_plugins().values()), key=attrgetter('name'))] - add_to_context_multi(plugins, color='yellow!') + if plugins: + info.append(cformat('*** %{magenta!}Plugins%{reset} ***')) + add_to_context_multi(plugins, color='yellow!') # Utils info.append(cformat('*** %{magenta!}Misc%{reset} ***')) add_to_context(celery, 'celery', doc='celery app', color='blue!') @@ -131,7 +132,6 @@ def shell_cmd(verbose, with_req_context): banner = '\n'.join(info + ['', banner]) ctx = current_app.make_shell_context() ctx.update(context) - clearCache() stack = ExitStack() if with_req_context: stack.enter_context(current_app.test_request_context(base_url=config.BASE_URL)) diff --git a/indico/cli/user.py b/indico/cli/user.py index 9b497cfb914..090bd11f99c 100644 --- a/indico/cli/user.py +++ b/indico/cli/user.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import click from flask_multipass import IdentityInfo from terminaltables import AsciiTable @@ -18,10 +16,6 @@ from indico.modules.users.operations import create_user from indico.modules.users.util import search_users from indico.util.console import cformat, prompt_email, prompt_pass -from indico.util.string import to_unicode - - -click.disable_unicode_literals_warning = True @cli_group() @@ -41,11 +35,11 @@ def _print_user_info(user): flags.append('%{cyan}pending%{reset}') print() print('User info:') - print(" ID: {}".format(user.id)) - print(" First name: {}".format(user.first_name)) - print(" Family name: {}".format(user.last_name)) - print(" Email: {}".format(user.email)) - print(" Affiliation: {}".format(user.affiliation)) + print(f" ID: {user.id}") + print(f" First name: {user.first_name}") + print(f" Family name: {user.last_name}") + print(f" Email: {user.email}") + print(f" Affiliation: {user.affiliation}") if flags: print(cformat(" Flags: {}".format(', '.join(flags)))) print() @@ -73,9 +67,9 @@ def _safe_lower(s): @click.option('--email', '-e', help='Email address of the user') @click.option('--affiliation', '-a', help='Affiliation of the user') def search(substring, include_deleted, include_pending, include_blocked, include_external, include_system, **criteria): - """Searches users matching some criteria""" - assert set(criteria.viewkeys()) == {'first_name', 'last_name', 'email', 'affiliation'} - criteria = {k: v for k, v in criteria.viewitems() if v is not None} + """Search users matching some criteria.""" + assert set(criteria.keys()) == {'first_name', 'last_name', 'email', 'affiliation'} + criteria = {k: v for k, v in criteria.items() if v is not None} res = search_users(exact=(not substring), include_deleted=include_deleted, include_pending=include_pending, include_blocked=include_blocked, external=include_external, allow_system_user=include_system, **criteria) @@ -83,7 +77,7 @@ def search(substring, include_deleted, include_pending, include_blocked, include print(cformat('%{yellow}No results found')) return elif len(res) > 100: - click.confirm('{} results found. Show them anyway?'.format(len(res)), abort=True) + click.confirm(f'{len(res)} results found. Show them anyway?', abort=True) users = sorted((u for u in res if isinstance(u, User)), key=lambda x: (x.first_name.lower(), x.last_name.lower(), x.email)) externals = sorted((ii for ii in res if isinstance(ii, IdentityInfo)), @@ -92,7 +86,7 @@ def search(substring, include_deleted, include_pending, include_blocked, include if users: table_data = [['ID', 'First Name', 'Last Name', 'Email', 'Affiliation']] for user in users: - table_data.append([unicode(user.id), user.first_name, user.last_name, user.email, user.affiliation]) + table_data.append([str(user.id), user.first_name, user.last_name, user.email, user.affiliation]) table = AsciiTable(table_data, cformat('%{white!}Users%{reset}')) table.justify_columns[0] = 'right' print(table.table) @@ -111,7 +105,7 @@ def search(substring, include_deleted, include_pending, include_blocked, include @cli.command() @click.option('--admin/--no-admin', '-a/', 'grant_admin', is_flag=True, help='Grant admin rights') def create(grant_admin): - """Creates a new user""" + """Create a new user.""" user_type = 'user' if not grant_admin else 'admin' while True: email = prompt_email() @@ -127,7 +121,7 @@ def create(grant_admin): print() while True: username = click.prompt("Enter username").lower().strip() - if not Identity.find(provider='indico', identifier=username).count(): + if not Identity.query.filter_by(provider='indico', identifier=username).has_rows(): break print(cformat('%{red}Username already exists')) password = prompt_pass() @@ -135,8 +129,7 @@ def create(grant_admin): return identity = Identity(provider='indico', identifier=username, password=password) - user = create_user(email, {'first_name': to_unicode(first_name), 'last_name': to_unicode(last_name), - 'affiliation': to_unicode(affiliation)}, identity) + user = create_user(email, {'first_name': first_name, 'last_name': last_name, 'affiliation': affiliation}, identity) user.is_admin = grant_admin _print_user_info(user) @@ -149,7 +142,7 @@ def create(grant_admin): @cli.command() @click.argument('user_id', type=int) def grant_admin(user_id): - """Grants administration rights to a given user""" + """Grant administration rights to a given user.""" user = User.get(user_id) if user is None: print(cformat("%{red}This user does not exist")) @@ -167,7 +160,7 @@ def grant_admin(user_id): @cli.command() @click.argument('user_id', type=int) def revoke_admin(user_id): - """Revokes administration rights from a given user""" + """Revoke administration rights from a given user.""" user = User.get(user_id) if user is None: print(cformat("%{red}This user does not exist")) @@ -185,7 +178,7 @@ def revoke_admin(user_id): @cli.command() @click.argument('user_id', type=int) def block(user_id): - """Blocks a given user""" + """Block a given user.""" user = User.get(user_id) if user is None: print(cformat("%{red}This user does not exist")) @@ -203,7 +196,7 @@ def block(user_id): @cli.command() @click.argument('user_id', type=int) def unblock(user_id): - """Unblocks a given user""" + """Unblock a given user.""" user = User.get(user_id) if user is None: print(cformat("%{red}This user does not exist")) diff --git a/indico/cli/util.py b/indico/cli/util.py index fc27e83b9a2..9b1a1e55880 100644 --- a/indico/cli/util.py +++ b/indico/cli/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import traceback from importlib import import_module @@ -25,7 +23,7 @@ def _create_app(info): from indico.web.flask.app import make_app - return make_app(set_path=True) + return make_app() class IndicoFlaskGroup(FlaskGroup): @@ -35,8 +33,8 @@ class IndicoFlaskGroup(FlaskGroup): """ def __init__(self, **extra): - super(IndicoFlaskGroup, self).__init__(create_app=_create_app, add_default_commands=False, - add_version_option=False, set_debug_flag=False, **extra) + super().__init__(create_app=_create_app, add_default_commands=False, add_version_option=False, + set_debug_flag=False, **extra) self._indico_plugin_commands = None def _load_plugin_commands(self): @@ -47,7 +45,7 @@ def _load_plugin_commands(self): def _wrap_in_plugin_context(self, plugin, cmd): cmd.callback = wrap_in_plugin_context(plugin, cmd.callback) - for subcmd in getattr(cmd, 'commands', {}).viewvalues(): + for subcmd in getattr(cmd, 'commands', {}).values(): self._wrap_in_plugin_context(plugin, subcmd) def _get_indico_plugin_commands(self, ctx): @@ -59,12 +57,12 @@ def _get_indico_plugin_commands(self, ctx): ctx.ensure_object(ScriptInfo).load_app() cmds = named_objects_from_signal(signals.plugin.cli.send(), plugin_attr='_indico_plugin') rv = {} - for name, cmd in cmds.viewitems(): + for name, cmd in cmds.items(): if cmd._indico_plugin: self._wrap_in_plugin_context(cmd._indico_plugin, cmd) rv[name] = cmd except Exception as exc: - if 'No indico config found' not in unicode(exc): + if 'No indico config found' not in str(exc): click.echo(click.style('Loading plugin commands failed:', fg='red', bold=True)) click.echo(click.style(traceback.format_exc(), fg='red')) rv = {} @@ -93,7 +91,7 @@ class LazyGroup(click.Group): def __init__(self, import_name, **kwargs): self._import_name = import_name - super(LazyGroup, self).__init__(**kwargs) + super().__init__(**kwargs) @cached_property def _impl(self): diff --git a/indico/cli/watchman.py b/indico/cli/watchman.py index 818e3c877c6..b248a74fb7c 100644 --- a/indico/cli/watchman.py +++ b/indico/cli/watchman.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import atexit import os import subprocess @@ -34,7 +32,7 @@ def _disable_reloader(argv): return argv -class Watcher(object): +class Watcher: def __init__(self, path, patterns): self.path = path self.name = path.replace('/', '-').strip('-') @@ -77,10 +75,10 @@ def check(self): return triggered def __repr__(self): - return ''.format(self.path, self.patterns) + return f'' -class Watchman(object): +class Watchman: def __init__(self): self._proc = None self._watchers = set() diff --git a/indico/core/auth.py b/indico/core/auth.py index 3ea9185ef48..fa6f0761018 100644 --- a/indico/core/auth.py +++ b/indico/core/auth.py @@ -1,26 +1,30 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals +import functools from flask import current_app, request from flask_multipass import InvalidCredentials, Multipass, NoSuchUser +from werkzeug.local import LocalProxy +from indico.core.config import config +from indico.core.limiter import make_rate_limiter from indico.core.logger import Logger logger = Logger.get('auth') +login_rate_limiter = LocalProxy(functools.cache(lambda: make_rate_limiter('login', config.FAILED_LOGIN_RATE_LIMIT))) class IndicoMultipass(Multipass): @property def default_local_auth_provider(self): """The default form-based auth provider.""" - return next((p for p in self.auth_providers.itervalues() if not p.is_external and p.settings.get('default')), + return next((p for p in self.auth_providers.values() if not p.is_external and p.settings.get('default')), None) @property @@ -29,11 +33,11 @@ def sync_provider(self): This is the identity provider used to sync user data. """ - return next((p for p in self.identity_providers.itervalues() if p.settings.get('synced_fields')), None) + return next((p for p in self.identity_providers.values() if p.settings.get('synced_fields')), None) @property def synced_fields(self): - """The keys to be synchronized + """The keys to be synchronized. This is the set of keys to be synced to user data. The ``email`` can never be synchronized. @@ -47,17 +51,17 @@ def synced_fields(self): return synced_fields def init_app(self, app): - super(IndicoMultipass, self).init_app(app) + super().init_app(app) with app.app_context(): self._check_default_provider() def _check_default_provider(self): # Ensure that there is maximum one sync provider - sync_providers = [p for p in self.identity_providers.itervalues() if p.settings.get('synced_fields')] + sync_providers = [p for p in self.identity_providers.values() if p.settings.get('synced_fields')] if len(sync_providers) > 1: raise ValueError('There can only be one sync provider.') # Ensure that there is exactly one form-based default auth provider - auth_providers = self.auth_providers.values() + auth_providers = list(self.auth_providers.values()) external_providers = [p for p in auth_providers if p.is_external] local_providers = [p for p in auth_providers if not p.is_external] if any(p.settings.get('default') for p in external_providers): @@ -75,6 +79,7 @@ def _check_default_provider(self): def handle_auth_error(self, exc, redirect_to_login=False): if isinstance(exc, (NoSuchUser, InvalidCredentials)): + login_rate_limiter.hit() logger.warning('Invalid credentials (ip=%s, provider=%s): %s', request.remote_addr, exc.provider.name if exc.provider else None, exc) else: @@ -84,7 +89,7 @@ def handle_auth_error(self, exc, redirect_to_login=False): fn = logger.debug fn('Authentication via %s failed: %s (%r)', exc.provider.name if exc.provider else None, exc_str, exc.details) - return super(IndicoMultipass, self).handle_auth_error(exc, redirect_to_login=redirect_to_login) + return super().handle_auth_error(exc, redirect_to_login=redirect_to_login) multipass = IndicoMultipass() diff --git a/indico/core/cache.py b/indico/core/cache.py index c39a6a4fc44..887952b1c49 100644 --- a/indico/core/cache.py +++ b/indico/core/cache.py @@ -1,11 +1,239 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. +from datetime import timedelta + from flask_caching import Cache +from flask_caching.backends.rediscache import RedisCache +from redis import RedisError +from redis import from_url as redis_from_url + +from indico.core.config import config +from indico.core.logger import Logger + + +_logger = Logger.get('cache') + + +class CachedNone: + __slots__ = () + + @classmethod + def wrap(cls, value): + return cls() if value is None else value + + @classmethod + def unwrap(cls, value, default=None): + if value is None: + return default + elif isinstance(value, cls): + return None + else: + return value + + +class IndicoRedisCache(RedisCache): + """ + This is similar to the original RedisCache from Flask-Caching, but it + allows specifying a default value when retrieving cache data and + distinguishing between a cached ``None`` value and a cache miss. + """ + + def dump_object(self, value): + # We are not overriding the `load_object` counterpart to this method o + # purpose because we need to have access to the wrapped value in `get` + # and `get_many`. + return super().dump_object(CachedNone.wrap(value)) + + def add(self, key, value, timeout=None): + # XXX: remove this once there's a release contining the fix from + # https://github.com/sh4nks/flask-caching/pull/218 + timeout = self._normalize_timeout(timeout) + dump = self.dump_object(value) + created = self._write_client.setnx( + name=self._get_prefix() + key, value=dump + ) + if created and timeout != -1: + self._write_client.expire( + name=self._get_prefix() + key, time=timeout + ) + return created + + def get(self, key, default=None): + return CachedNone.unwrap(super().get(key), default) + + def get_many(self, *keys, default=None): + return [CachedNone.unwrap(val, default) for val in super().get_many(*keys)] + + def get_dict(self, *keys, default=None): + return dict(zip(keys, self.get_many(*keys, default=default))) + + @classmethod + def factory(cls, app, config, args, kwargs): + key_prefix = config.get('CACHE_KEY_PREFIX') + if key_prefix: + kwargs['key_prefix'] = key_prefix + kwargs['host'] = redis_from_url(config['CACHE_REDIS_URL'], socket_timeout=1) + return IndicoRedisCache(*args, **kwargs) + + +class ScopedCache: + def __init__(self, cache, scope): + self.cache = cache + self.scope = scope + + def _scoped(self, key): + return f'{self.scope}/{key}' + + def get(self, key, default=None): + return self.cache.get(self._scoped(key), default=default) + + def set(self, key, value, timeout=None): + if isinstance(timeout, timedelta): + timeout = int(timeout.total_seconds()) + self.cache.set(self._scoped(key), value, timeout=timeout) + + def add(self, key, value, timeout=None): + if isinstance(timeout, timedelta): + timeout = int(timeout.total_seconds()) + self.cache.add(self._scoped(key), value, timeout=timeout) + + def delete(self, key): + self.cache.delete(self._scoped(key)) + + def delete_many(self, *keys): + keys = [self._scoped(key) for key in keys] + self.cache.delete_many(*keys) + + def clear(self): + raise NotImplementedError('Clearing scoped caches is not supported') + + def get_dict(self, *keys, default=None): + return dict(zip(keys, self.get_many(*keys, default=default))) + + def get_many(self, *keys, default=None): + keys = [self._scoped(key) for key in keys] + return self.cache.get_many(*keys, default=default) + + def set_many(self, mapping, timeout=None): + if isinstance(timeout, timedelta): + timeout = int(timeout.total_seconds()) + mapping = {self._scoped(key): value for key, value in mapping.items()} + self.cache.set_many(mapping, timeout=timeout) + + def __repr__(self): + return f'' + + +class IndicoCache(Cache): + """ + This is basicaly the Cache class from Flask-Caching but it silences all + exceptions that happen during a cache operation since cache failures should + not take down the whole page. + + While this cache can in principle support many different backends, we only + consider redis and (for unittests) a simple dict-based cache. This allows + us to be more specific in catching exceptions since the Redis cache has + exactly one base exception. + """ + + def get(self, key, default=None): + try: + return super().get(key, default) + except RedisError: + if config.DEBUG: + raise + _logger.exception('get(%r) failed', key) + return default + + def set(self, key, value, timeout=None): + if isinstance(timeout, timedelta): + timeout = int(timeout.total_seconds()) + try: + super().set(key, value, timeout=timeout) + except RedisError: + if config.DEBUG: + raise + _logger.exception('set(%r) failed', key) + + def add(self, key, value, timeout=None): + if isinstance(timeout, timedelta): + timeout = int(timeout.total_seconds()) + try: + super().add(key, value, timeout=timeout) + except RedisError: + if config.DEBUG: + raise + _logger.exception('add(%r) failed', key) + + def delete(self, key): + try: + super().delete(key) + except RedisError: + if config.DEBUG: + raise + _logger.exception('delete(%r) failed', key) + + def delete_many(self, *keys): + try: + super().delete_many(*keys) + except RedisError: + if config.DEBUG: + raise + _logger.exception('delete_many(%s) failed', ', '.join(map(repr, keys))) + + def clear(self): + try: + super().clear() + except RedisError: + if config.DEBUG: + raise + _logger.exception('clear() failed') + + def get_many(self, *keys, default=None): + try: + return super().get_many(*keys, default=default) + except RedisError: + if config.DEBUG: + raise + logkeys = ', '.join(map(repr, keys)) + _logger.exception('get_many(%s) failed', logkeys) + return [default] * len(keys) + + def set_many(self, mapping, timeout=None): + if isinstance(timeout, timedelta): + timeout = int(timeout.total_seconds()) + try: + super().set_many(mapping, timeout=timeout) + except RedisError: + if config.DEBUG: + raise + _logger.exception('set_many(%r) failed', mapping) + + def get_dict(self, *keys, default=None): + try: + return super().get_dict(*keys, default=default) + except RedisError: + if config.DEBUG: + raise + logkeys = ', '.join(map(repr, keys)) + _logger.exception('get_dict(%s) failed', logkeys) + return dict(zip(keys, [default] * len(keys))) + + +def make_scoped_cache(scope): + """Create a new scoped cache. + + In most cases the global cache should not be used directly but rather + with a scope depending on the module a cache is used for. This is + especially important when passing user-provided data as the cache key + to prevent reading other unrelated cache keys. + """ + return ScopedCache(cache, scope) -cache = Cache() +cache = IndicoCache() diff --git a/indico/core/cache_test.py b/indico/core/cache_test.py new file mode 100644 index 00000000000..6642943d3ee --- /dev/null +++ b/indico/core/cache_test.py @@ -0,0 +1,71 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from datetime import timedelta + +import pytest + +from indico.core.cache import cache, make_scoped_cache + + +def test_cache_none_default(): + assert cache.get('foo') is None + assert cache.get('foo', 'bar') == 'bar' + cache.set('foo', None) + cache.set('bar', 0) + cache.add('foobar', None) + cache.add('foobar', 'nope') + assert cache.get('foo') is None + assert cache.get('foo', 'bar') is None + assert cache.get('foobar') is None + assert cache.get('foobar', 'bar') is None + assert cache.get_dict('foo', 'bar', 'foobar') == {'foo': None, 'bar': 0, 'foobar': None} + assert cache.get_dict('foo', 'bar', 'foobar', default='x') == {'foo': None, 'bar': 0, 'foobar': None} + assert cache.get_many('foo', 'bar', 'foobar') == [None, 0, None] + assert cache.get_many('foo', 'bar', 'foobar', default='x') == [None, 0, None] + + +def test_scoped_cache(): + cache.set('foo', 1) + cache.set('foobar', 2) + scoped = make_scoped_cache('test') + assert scoped.get('foo', 'notset') == 'notset' + + scoped.set('foo', 'bar') + scoped.add('foobar', 'test') + scoped.add('foo', 'nope') + + # accessing the scope through the global cache is possibly, but should not be done + # if this ever starts failing because we change something in the cache implementation + # removing this particular assertion is fine + assert cache.get('test/foo') == 'bar' + + assert scoped.get('foo') == 'bar' + assert scoped.get_many('foo', 'foobar') == ['bar', 'test'] + assert scoped.get_dict('foo', 'foobar') == {'foo': 'bar', 'foobar': 'test'} + + scoped.delete('foobar') + scoped.delete_many('foo') + assert scoped.get_many('foo', 'foobar') == [None, None] + + # ensure deletions only affected the scoped keys + assert cache.get_many('foo', 'foobar') == [1, 2] + + scoped.set_many({'a': 'aa', 'b': 'bb'}) + assert scoped.get_dict('a', 'b') == {'a': 'aa', 'b': 'bb'} + + assert cache.get_many('foo', 'foobar', 'a', 'b') == [1, 2, None, None] + + +@pytest.mark.parametrize('scoped', (False, True)) +@pytest.mark.parametrize('timeout', (5, timedelta(seconds=5))) +def test_expiry(scoped, timeout): + cache_obj = make_scoped_cache('test') if scoped else cache + cache_obj.set('a', 1, timeout=timeout) + cache_obj.add('b', 2, timeout=timeout) + cache_obj.set_many({'c': 3}, timeout=timeout) + assert cache_obj.get_many('a', 'b', 'c') == [1, 2, 3] diff --git a/indico/core/celery/__init__.py b/indico/core/celery/__init__.py index 413be43ee83..dd03e9cf8fc 100644 --- a/indico/core/celery/__init__.py +++ b/indico/core/celery/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import timedelta from celery.schedules import crontab @@ -27,11 +25,12 @@ from indico.web.menu import SideMenuItem -__all__ = ('celery',) +__all__ = ('celery', 'AsyncResult') #: The Celery instance for all Indico tasks celery = IndicoCelery('indico') +AsyncResult = celery.AsyncResult celery_settings = SettingsProxy('celery', { diff --git a/indico/core/celery/blueprint.py b/indico/core/celery/blueprint.py index 401283755cb..1cc2c0e644e 100644 --- a/indico/core/celery/blueprint.py +++ b/indico/core/celery/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.celery.controllers import RHCeleryTasks from indico.web.flask.wrappers import IndicoBlueprint diff --git a/indico/core/celery/cli.py b/indico/core/celery/cli.py index 7d2d88f2af0..42ed3a83e4f 100644 --- a/indico/core/celery/cli.py +++ b/indico/core/celery/cli.py @@ -1,68 +1,24 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals +import click +from celery.bin.celery import celery as celery_cmd -import os -import sys - -from celery.bin.base import Command -from celery.bin.celery import CeleryCommand, command_classes - -from indico.core.celery import celery from indico.core.celery.util import unlock_task -from indico.core.config import config -from indico.modules.oauth.models.applications import OAuthApplication, SystemAppType from indico.util.console import cformat -from indico.web.flask.util import url_for - - -def celery_cmd(args): - # remove the celery shell command - next(funcs for group, funcs, _ in command_classes if group == 'Main').remove('shell') - del CeleryCommand.commands['shell'] - if args and args[0] == 'flower': - # Somehow flower hangs when executing it using CeleryCommand() so we simply exec it directly. - # It doesn't really need the celery config anyway (besides the broker url) - try: - import flower # noqa: F401 - except ImportError: - print(cformat('%{red!}Flower is not installed')) - sys.exit(1) +# remove the celery shell command +del celery_cmd.commands['shell'] - app = OAuthApplication.find_one(system_app_type=SystemAppType.flower) - if not app.redirect_uris: - print(cformat('%{yellow!}Authentication will fail unless you configure the redirect url for the {} OAuth ' - 'application in the administration area.').format(app.name)) - print(cformat('%{green!}Only Indico admins will have access to flower.')) - print(cformat('%{yellow}Note that revoking admin privileges will not revoke Flower access.')) - print(cformat('%{yellow}To force re-authentication, restart Flower.')) - auth_args = ['--auth=^Indico Admin$', '--auth_provider=indico.core.celery.flower.FlowerAuthHandler'] - auth_env = {'INDICO_FLOWER_CLIENT_ID': app.client_id, - 'INDICO_FLOWER_CLIENT_SECRET': app.client_secret, - 'INDICO_FLOWER_AUTHORIZE_URL': url_for('oauth.oauth_authorize', _external=True), - 'INDICO_FLOWER_TOKEN_URL': url_for('oauth.oauth_token', _external=True), - 'INDICO_FLOWER_USER_URL': url_for('users.authenticated_user', _external=True)} - if config.FLOWER_URL: - auth_env['INDICO_FLOWER_URL'] = config.FLOWER_URL - args = ['celery', '-b', config.CELERY_BROKER] + args + auth_args - env = dict(os.environ, **auth_env) - os.execvpe('celery', args, env) - elif args and args[0] == 'shell': - print(cformat('%{red!}Please use `indico shell`.')) - sys.exit(1) - else: - CeleryCommand(celery).execute_from_commandline(['indico celery'] + args) - - -class UnlockCommand(Command): +@celery_cmd.command() +@click.argument('name') +def unlock(name): """Unlock a locked task. Use this if your celery worker was e.g. killed by your kernel's @@ -73,8 +29,7 @@ class UnlockCommand(Command): indico celery unlock event_reminders """ - def run(self, name, **kwargs): - if unlock_task(name): - print(cformat('%{green!}Task {} unlocked').format(name)) - else: - print(cformat('%{yellow}Task {} is not locked').format(name)) + if unlock_task(name): + print(cformat('%{green!}Task {} unlocked').format(name)) + else: + print(cformat('%{yellow}Task {} is not locked').format(name)) diff --git a/indico/core/celery/controllers.py b/indico/core/celery/controllers.py index 1964cb52871..4e64ac2f6c6 100644 --- a/indico/core/celery/controllers.py +++ b/indico/core/celery/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import timedelta from operator import itemgetter diff --git a/indico/core/celery/core.py b/indico/core/celery/core.py index bcd58bef90c..d98f15630c6 100644 --- a/indico/core/celery/core.py +++ b/indico/core/celery/core.py @@ -1,20 +1,18 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import logging import os +from contextlib import ExitStack from operator import itemgetter from celery import Celery from celery.app.log import Logging from celery.beat import PersistentScheduler -from contextlib2 import ExitStack from flask_pluginengine import current_plugin, plugin_context from sqlalchemy import inspect from terminaltables import AsciiTable @@ -25,13 +23,11 @@ from indico.core.notifications import flush_email_queue, init_email_queue from indico.core.plugins import plugin_engine from indico.util.console import cformat -from indico.util.fossilize import clearCache -from indico.util.string import return_ascii from indico.web.flask.stats import request_stats_request_started class IndicoCelery(Celery): - """Celery sweetened with some Indico/Flask-related sugar + """Celery sweetened with some Indico/Flask-related sugar. The following extra params are available on the `task` decorator: @@ -45,7 +41,7 @@ class IndicoCelery(Celery): def __init__(self, *args, **kwargs): kwargs.setdefault('log', IndicoCeleryLogging) - super(IndicoCelery, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.flask_app = None self._patch_task() @@ -65,6 +61,7 @@ def init_app(self, app): self.conf['result_serializer'] = 'pickle' self.conf['task_serializer'] = 'pickle' self.conf['accept_content'] = ['json', 'yaml', 'pickle'] + self.conf['task_always_eager'] = app.config['TESTING'] # Allow indico.conf to override settings self.conf.update(config.CELERY_CONFIG) assert self.flask_app is None or self.flask_app is app @@ -100,7 +97,7 @@ def decorator(f): return decorator def _patch_task(self): - """Patches the `task` decorator to run tasks inside the indico environment""" + """Patch the `task` decorator to run tasks inside the indico environment.""" class IndicoTask(self.Task): abstract = True @@ -113,8 +110,8 @@ def apply_async(s, args=None, kwargs=None, task_id=None, producer=None, if current_plugin: options['headers'] = options.get('headers') or {} # None in a retry options['headers']['indico_plugin'] = current_plugin.name - return super(IndicoTask, s).apply_async(args=args, kwargs=kwargs, task_id=task_id, producer=producer, - link=link, link_error=link_error, shadow=shadow, **options) + return super().apply_async(args=args, kwargs=kwargs, task_id=task_id, producer=producer, + link=link, link_error=link_error, shadow=shadow, **options) def __call__(s, *args, **kwargs): stack = ExitStack() @@ -124,18 +121,17 @@ def __call__(s, *args, **kwargs): args = _CelerySAWrapper.unwrap_args(args) kwargs = _CelerySAWrapper.unwrap_kwargs(kwargs) plugin = getattr(s, 'plugin', s.request.get('indico_plugin')) - if isinstance(plugin, basestring): + if isinstance(plugin, str): plugin_name = plugin plugin = plugin_engine.get_plugin(plugin) if plugin is None: stack.close() raise ValueError('Plugin not active: ' + plugin_name) stack.enter_context(plugin_context(plugin)) - clearCache() with stack: request_stats_request_started() init_email_queue() - rv = super(IndicoTask, s).__call__(*args, **kwargs) + rv = super().__call__(*args, **kwargs) flush_email_queue() return rv @@ -147,15 +143,15 @@ def _configure_logger(self, logger, *args, **kwargs): # don't let celery mess with the root logger if logger is logging.getLogger(): return - super(IndicoCeleryLogging, self)._configure_logger(logger, *args, **kwargs) + super()._configure_logger(logger, *args, **kwargs) class IndicoPersistentScheduler(PersistentScheduler): - """Celery scheduler that allows indico.conf to override specific entries""" + """Celery scheduler that allows indico.conf to override specific entries.""" def setup_schedule(self): deleted = set() - for task_name, entry in config.SCHEDULED_TASK_OVERRIDE.iteritems(): + for task_name, entry in config.SCHEDULED_TASK_OVERRIDE.items(): if task_name not in self.app.conf['beat_schedule']: self.logger.error('Invalid entry in ScheduledTaskOverride: %s', task_name) continue @@ -167,7 +163,7 @@ def setup_schedule(self): self.app.conf['beat_schedule'][task_name].update(entry) else: self.app.conf['beat_schedule'][task_name]['schedule'] = entry - super(IndicoPersistentScheduler, self).setup_schedule() + super().setup_schedule() if not self.app.conf['worker_redirect_stdouts']: # print the schedule unless we are in production where # this output would get redirected to a logger which is @@ -176,7 +172,7 @@ def setup_schedule(self): def _print_schedule(self, deleted): table_data = [['Name', 'Schedule']] - for entry in sorted(self.app.conf['beat_schedule'].itervalues(), key=itemgetter('task')): + for entry in sorted(self.app.conf['beat_schedule'].values(), key=itemgetter('task')): table_data.append([cformat('%{yellow!}{}%{reset}').format(entry['task']), cformat('%{green}{!r}%{reset}').format(entry['schedule'])]) for task_name in sorted(deleted): @@ -185,7 +181,7 @@ def _print_schedule(self, deleted): print(AsciiTable(table_data, cformat('%{white!}Periodic Tasks%{reset}')).table) -class _CelerySAWrapper(object): +class _CelerySAWrapper: """Wrapper to safely pass SQLAlchemy objects to tasks. This is achieved by passing only the model name and its PK values @@ -204,10 +200,9 @@ def __init__(self, obj): def object(self): obj = self.identity_key[0].get(self.identity_key[1]) if obj is None: - raise ValueError('Object not in DB: {}'.format(self)) + raise ValueError(f'Object not in DB: {self}') return obj - @return_ascii def __repr__(self): model, args = self.identity_key[:2] return '<{}: {}>'.format(model.__name__, ','.join(map(repr, args))) @@ -218,7 +213,7 @@ def wrap_args(cls, args): @classmethod def wrap_kwargs(cls, kwargs): - return {k: cls(v) if isinstance(v, db.Model) else v for k, v in kwargs.iteritems()} + return {k: cls(v) if isinstance(v, db.Model) else v for k, v in kwargs.items()} @classmethod def unwrap_args(cls, args): @@ -226,4 +221,4 @@ def unwrap_args(cls, args): @classmethod def unwrap_kwargs(cls, kwargs): - return {k: v.object if isinstance(v, cls) else v for k, v in kwargs.iteritems()} + return {k: v.object if isinstance(v, cls) else v for k, v in kwargs.items()} diff --git a/indico/core/celery/flower.py b/indico/core/celery/flower.py deleted file mode 100644 index 8a4764c3bc6..00000000000 --- a/indico/core/celery/flower.py +++ /dev/null @@ -1,102 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from __future__ import absolute_import, unicode_literals - -import functools -import json -import os -from urllib import urlencode - -from flower.urls import settings -from flower.views import BaseHandler -from tornado.auth import AuthError, OAuth2Mixin, _auth_return_future -from tornado.httpclient import AsyncHTTPClient, HTTPClient, HTTPRequest -from tornado.options import options -from tornado.web import HTTPError, asynchronous - - -class FlowerAuthHandler(BaseHandler, OAuth2Mixin): - _OAUTH_NO_CALLBACKS = False - - @property - def _OAUTH_AUTHORIZE_URL(self): - return os.environ['INDICO_FLOWER_AUTHORIZE_URL'] - - @property - def _OAUTH_ACCESS_TOKEN_URL(self): - return os.environ['INDICO_FLOWER_TOKEN_URL'] - - @_auth_return_future - def get_authenticated_user(self, redirect_uri, code, callback): - http = self.get_auth_http_client() - body = urlencode({ - 'redirect_uri': redirect_uri, - 'code': code, - 'client_id': os.environ['INDICO_FLOWER_CLIENT_ID'], - 'client_secret': os.environ['INDICO_FLOWER_CLIENT_SECRET'], - 'grant_type': 'authorization_code', - }) - http.fetch(self._OAUTH_ACCESS_TOKEN_URL, - functools.partial(self._on_access_token, callback), - method='POST', - headers={'Content-Type': 'application/x-www-form-urlencoded', 'Accept': 'application/json'}, - body=body, - validate_cert=False) - - @asynchronous - def _on_access_token(self, future, response): - if response.error: - future.set_exception(AuthError('OAuth authentication error: {}'.format(response))) - return - future.set_result(json.loads(response.body)) - - def get_auth_http_client(self): - return AsyncHTTPClient() - - @property - def uri_base(self): - try: - return os.environ['INDICO_FLOWER_URL'] - except KeyError: - return 'http{}://{}{}{}'.format('s' if 'ssl_options' in settings else '', - options.address or 'localhost', - ':{}'.format(options.port) if not options.unix_socket else '', - '/{}'.format(options.url_prefix) if options.url_prefix else '') - - @asynchronous - def get(self): - redirect_uri = '{}/login'.format(self.uri_base.rstrip('/')) - if self.get_argument('code', False): - self.get_authenticated_user( - redirect_uri=redirect_uri, - code=self.get_argument('code'), - callback=self._on_auth, - ) - else: - self.authorize_redirect( - redirect_uri=redirect_uri, - client_id=os.environ['INDICO_FLOWER_CLIENT_ID'], - scope=['read:user'], - response_type='code', - extra_params={'approval_prompt': 'auto'} - ) - - @asynchronous - def _on_auth(self, user): - if not user: - raise HTTPError(500, 'OAuth authentication failed') - access_token = user['access_token'] - req = HTTPRequest(os.environ['INDICO_FLOWER_USER_URL'], - headers={'Authorization': 'Bearer ' + access_token, 'User-agent': 'Tornado auth'}, - validate_cert=False) - response = HTTPClient().fetch(req) - payload = json.loads(response.body.decode('utf-8')) - if not payload or not payload['admin']: - raise HTTPError(403, 'Access denied') - self.set_secure_cookie('user', 'Indico Admin') - self.redirect(self.get_argument('next', self.uri_base)) diff --git a/indico/core/celery/templates/celery_tasks.html b/indico/core/celery/templates/celery_tasks.html index 1ddf53a9370..8cbe73d7175 100644 --- a/indico/core/celery/templates/celery_tasks.html +++ b/indico/core/celery/templates/celery_tasks.html @@ -1,5 +1,4 @@ {% extends 'layout/admin_page.html' %} -{% from 'message_box.html' import message_box %} {% block title %} {% trans %}Tasks{% endtrans %} @@ -7,14 +6,6 @@ {% block content %}
- {%- if indico_config.FLOWER_URL -%} - {%- call message_box('info', fixed_width=true) -%} - {%- trans flower_url=indico_config.FLOWER_URL -%} - Monitor Indico's Celery tasks with Flower - here. - {%- endtrans -%} - {%- endcall %} - {%- endif -%}

{% trans %}Periodic Tasks{% endtrans %}

diff --git a/indico/core/celery/util.py b/indico/core/celery/util.py index 674a38b2866..21830611990 100644 --- a/indico/core/celery/util.py +++ b/indico/core/celery/util.py @@ -1,36 +1,36 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import wraps from celery import current_task +from indico.core.cache import make_scoped_cache from indico.core.logger import Logger -from indico.legacy.common.cache import GenericCache + + +_lock_cache = make_scoped_cache('task-locks') def locked_task(f): """Decorator to prevent a task from running multiple times at once.""" @wraps(f) def wrapper(*args, **kwargs): - cache = GenericCache('task-locks') name = current_task.name - if cache.get(name): + if _lock_cache.get(name): Logger.get('celery').warning('Task %s is locked; not executing it. ' 'To manually unlock it, run `indico celery unlock %s`', name, name) return - cache.set(name, True, 86400) + _lock_cache.set(name, True, 86400) try: return f(*args, **kwargs) finally: - cache.delete(name) + _lock_cache.delete(name) return wrapper @@ -39,8 +39,7 @@ def unlock_task(name): :return: ``True`` if the task has been unlocked; ``False`` if it was not locked. """ - cache = GenericCache('task-locks') - if not cache.get(name): + if not _lock_cache.get(name): return False - cache.delete(name) + _lock_cache.delete(name) return True diff --git a/indico/core/celery/views.py b/indico/core/celery/views.py index 48b3fba652c..e1a09795f66 100644 --- a/indico/core/celery/views.py +++ b/indico/core/celery/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.admin.views import WPAdmin diff --git a/indico/core/config.py b/indico/core/config.py index 33cc46f505c..cb9e998329d 100644 --- a/indico/core/config.py +++ b/indico/core/config.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import, unicode_literals - import ast import codecs import os @@ -32,7 +30,6 @@ 'ATTACHMENT_STORAGE': 'default', 'AUTH_PROVIDERS': {}, 'BASE_URL': None, - 'CACHE_BACKEND': 'files', 'CACHE_DIR': '/opt/indico/cache', 'CATEGORY_CLEANUP': {}, 'CELERY_BROKER': None, @@ -49,9 +46,10 @@ 'DEFAULT_TIMEZONE': 'UTC', 'DISABLE_CELERY_CHECK': None, 'ENABLE_ROOMBOOKING': False, + 'EXPERIMENTAL_EDITING_SERVICE': False, 'EXTERNAL_REGISTRATION_URL': None, - 'FLOWER_URL': None, 'HELP_URL': 'https://learn.getindico.io', + 'FAILED_LOGIN_RATE_LIMIT': '5 per 15 minutes; 10 per day', 'IDENTITY_PROVIDERS': {}, 'LOCAL_IDENTITIES': True, 'LOCAL_MODERATION': False, @@ -143,9 +141,9 @@ def _parse_config(path): locals_ = {} with codecs.open(path, encoding='utf-8') as config_file: # XXX: unicode_literals is inherited from this file - exec compile(config_file.read(), path, 'exec') in globals_, locals_ - return {unicode(k if k.isupper() else _convert_key(k)): v - for k, v in locals_.iteritems() + exec(compile(config_file.read(), path, 'exec'), globals_, locals_) + return {str(k if k.isupper() else _convert_key(k)): v + for k, v in locals_.items() if k[0] != '_'} @@ -160,8 +158,6 @@ def _convert_key(name): def _postprocess_config(data): - if data['DEFAULT_TIMEZONE'] not in pytz.all_timezones_set: - raise ValueError('Invalid default timezone: {}'.format(data['DEFAULT_TIMEZONE'])) data['BASE_URL'] = data['BASE_URL'].rstrip('/') data['STATIC_SITE_STORAGE'] = data['STATIC_SITE_STORAGE'] or data['ATTACHMENT_STORAGE'] if data['DISABLE_CELERY_CHECK'] is None: @@ -173,8 +169,8 @@ def _sanitize_data(data, allow_internal=False): if allow_internal: allowed |= set(INTERNAL_DEFAULTS) for key in set(data) - allowed: - warnings.warn('Ignoring unknown config key {}'.format(key)) - return {k: v for k, v in data.iteritems() if k in allowed} + warnings.warn(f'Ignoring unknown config key {key}') + return {k: v for k, v in data.items() if k in allowed} def load_config(only_defaults=False, override=None): @@ -208,7 +204,7 @@ def load_config(only_defaults=False, override=None): return ImmutableDict(data) -class IndicoConfig(object): +class IndicoConfig: """Wrapper for the Indico configuration. It exposes all config keys as read-only attributes. @@ -255,12 +251,18 @@ def CONFERENCE_CSS_TEMPLATES_BASE_URL(self): @property def IMAGES_BASE_URL(self): - return 'static/images' if g.get('static_site') else url_parse('{}/images'.format(self.BASE_URL)).path + return 'static/images' if g.get('static_site') else url_parse(f'{self.BASE_URL}/images').path @property def LATEX_ENABLED(self): return bool(self.XELATEX_PATH) + def validate(self): + from indico.core.auth import login_rate_limiter + login_rate_limiter._get_current_object() # fail in case FAILED_LOGIN_RATE_LIMIT invalid + if self.DEFAULT_TIMEZONE not in pytz.all_timezones_set: + raise ValueError(f'Invalid default timezone: {self.DEFAULT_TIMEZONE}') + def __getattr__(self, name): try: return self.data[name] diff --git a/indico/core/db/__init__.py b/indico/core/db/__init__.py index 2be50ffd7c5..65dbaded5e9 100644 --- a/indico/core/db/__init__.py +++ b/indico/core/db/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import - from .sqlalchemy import db diff --git a/indico/core/db/sqlalchemy/__init__.py b/indico/core/db/sqlalchemy/__init__.py index b45bde13f5d..24241072b7d 100644 --- a/indico/core/db/sqlalchemy/__init__.py +++ b/indico/core/db/sqlalchemy/__init__.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/db/sqlalchemy/attachments.py b/indico/core/db/sqlalchemy/attachments.py index fd8449be941..d0ab1c16e3b 100644 --- a/indico/core/db/sqlalchemy/attachments.py +++ b/indico/core/db/sqlalchemy/attachments.py @@ -1,22 +1,23 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy import orm from sqlalchemy.event import listens_for from indico.core.db import db +from indico.core.db.sqlalchemy.util.models import get_all_models from indico.util.caching import memoize_request -class AttachedItemsMixin(object): - """Allows for easy retrieval of structured information about - items attached to the object""" +class AttachedItemsMixin: + """ + Allow for easy retrieval of structured information about + items attached to the object. + """ #: When set to ``True`` will preload all items that exist for the same event. #: Should be set to False when not applicable (no object.event[_new] property). @@ -50,7 +51,8 @@ def _make_attachment_count_column_property(cls): ~Attachment.is_deleted, (getattr(AttachmentFolder, cls.ATTACHMENT_FOLDER_ID_COLUMN) == cls.id) )) - .correlate_except(AttachmentFolder, Attachment)) + .correlate_except(AttachmentFolder, Attachment) + .scalar_subquery()) cls.attachment_count = db.column_property(query, deferred=True) @@ -58,6 +60,6 @@ def _make_attachment_count_column_property(cls): def _mappers_configured(): # We need to create the column property here since we cannot import # Attachment/AttachmentFolder while the models are being defined - for model in db.Model._decl_class_registry.itervalues(): + for model in get_all_models(): if hasattr(model, '__table__') and issubclass(model, AttachedItemsMixin): _make_attachment_count_column_property(model) diff --git a/indico/core/db/sqlalchemy/colors.py b/indico/core/db/sqlalchemy/colors.py index 860c77b96cc..7bae0cb0f61 100644 --- a/indico/core/db/sqlalchemy/colors.py +++ b/indico/core/db/sqlalchemy/colors.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import string from collections import namedtuple @@ -42,17 +40,17 @@ def __new__(cls, text, background): raise ValueError('Colors must be be `rgb` or `rrggbb`') if not all(c in string.hexdigits for color in colors for c in color): raise ValueError('Colors must only use hex digits') - return super(ColorTuple, cls).__new__(cls, *colors) + return super().__new__(cls, *colors) def __nonzero__(self): return all(self) @property def css(self): - return 'color: #{} !important; background: #{} !important'.format(self.text, self.background) + return f'color: #{self.text} !important; background: #{self.background} !important' -class ColorMixin(object): +class ColorMixin: """Mixin to store text+background colors in a model. For convenience (e.g. for WTForms integrations when selecting both @@ -95,7 +93,7 @@ def background_color(cls): @property def colors(self): - """The current set of colors or None if no colors are set""" + """The current set of colors or None if no colors are set.""" colors = ColorTuple(self.text_color, self.background_color) return colors or self.default_colors diff --git a/indico/core/db/sqlalchemy/core.py b/indico/core/db/sqlalchemy/core.py index 1814c9ec9d6..91caa00ff92 100644 --- a/indico/core/db/sqlalchemy/core.py +++ b/indico/core/db/sqlalchemy/core.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import - import sys from contextlib import contextmanager from functools import partial @@ -27,14 +25,14 @@ class ConstraintViolated(Exception): - """Indicates that a constraint trigger was violated""" + """Indicate that a constraint trigger was violated.""" def __init__(self, message, orig): - super(ConstraintViolated, self).__init__(message) + super().__init__(message) self.orig = orig def handle_sqlalchemy_database_error(): - """Handle a SQLAlchemy DatabaseError exception nicely + """Handle a SQLAlchemy DatabaseError exception nicely. Currently this only takes care of custom INDX exception raised from constraint triggers. It must be invoked from an @@ -52,10 +50,10 @@ def handle_sqlalchemy_database_error(): raise msg = exc.orig.diag.message_primary if exc.orig.diag.message_detail: - msg += ': {}'.format(exc.orig.diag.message_detail) + msg += f': {exc.orig.diag.message_detail}' if exc.orig.diag.message_hint: - msg += ' ({})'.format(exc.orig.diag.message_hint) - raise ConstraintViolated(msg, exc.orig), None, tb # raise with original traceback + msg += f' ({exc.orig.diag.message_hint})' + raise ConstraintViolated(msg, exc.orig) from exc def _after_commit(*args, **kwargs): @@ -66,16 +64,16 @@ def _after_commit(*args, **kwargs): class IndicoSQLAlchemy(SQLAlchemy): def __init__(self, *args, **kwargs): - super(IndicoSQLAlchemy, self).__init__(*args, **kwargs) - self.m = type(b'_Models', (object,), {}) + super().__init__(*args, **kwargs) + self.m = type('_Models', (object,), {}) def create_session(self, *args, **kwargs): - session = super(IndicoSQLAlchemy, self).create_session(*args, **kwargs) + session = super().create_session(*args, **kwargs) listen(session, 'after_commit', _after_commit) return session def enforce_constraints(self): - """Enables immedaite enforcing of deferred constraints. + """Enable immediate enforcing of deferred constraints. This should be done at the end of normal request processing and exceptions should be handled in a clean way that goes @@ -98,7 +96,7 @@ def enforce_constraints(self): @contextmanager def tmp_session(self): - """Provides a contextmanager with a temporary SQLAlchemy session. + """Provide a contextmanager with a temporary SQLAlchemy session. This allows you to use SQLAlchemy e.g. in a `after_this_request` callback without having to worry about things like the ZODB extension, @@ -133,7 +131,7 @@ def _before_create(target, connection, **kw): for schema in schemas: if not _schema_exists(connection, schema): CreateSchema(schema).execute(connection) - signals.db_schema_created.send(unicode(schema), connection=connection) + signals.db_schema_created.send(str(schema), connection=connection) # Create our custom functions create_unaccent_function(connection) create_natsort_function(connection) @@ -156,7 +154,7 @@ def _coerce_custom(target, value, oldvalue, initiator, fn): def _column_names(constraint, table): - return '_'.join((c if isinstance(c, basestring) else c.name) for c in constraint.columns) + return '_'.join((c if isinstance(c, str) else c.name) for c in constraint.columns) def _unique_index(constraint, table): diff --git a/indico/core/db/sqlalchemy/custom/__init__.py b/indico/core/db/sqlalchemy/custom/__init__.py index 24198f18a2e..90ed471fa62 100644 --- a/indico/core/db/sqlalchemy/custom/__init__.py +++ b/indico/core/db/sqlalchemy/custom/__init__.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/db/sqlalchemy/custom/greatest.py b/indico/core/db/sqlalchemy/custom/greatest.py index 4bd7a2afcba..00a626e37d5 100644 --- a/indico/core/db/sqlalchemy/custom/greatest.py +++ b/indico/core/db/sqlalchemy/custom/greatest.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/db/sqlalchemy/custom/int_enum.py b/indico/core/db/sqlalchemy/custom/int_enum.py index edc3c8f2b9b..da6d52fd0cb 100644 --- a/indico/core/db/sqlalchemy/custom/int_enum.py +++ b/indico/core/db/sqlalchemy/custom/int_enum.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import textwrap from sqlalchemy import type_coerce @@ -43,7 +41,7 @@ class PyIntEnum(TypeDecorator, SchemaType): def __init__(self, enum=None, exclude_values=None): self.enum = enum - self.exclude_values = set(exclude_values or ()) + self.exclude_values = frozenset(exclude_values or ()) TypeDecorator.__init__(self) SchemaType.__init__(self) @@ -67,8 +65,8 @@ def coerce_set_value(self, value): return self.enum(value) def alembic_render_type(self, autogen_context, toplevel_code): - name = '_{}'.format(self.enum.__name__) - members = '\n'.join(' {} = {!r}'.format(x.name, x.value) for x in self.enum) + name = f'_{self.enum.__name__}' + members = '\n'.join(f' {x.name} = {x.value!r}' for x in self.enum) enum_tpl = textwrap.dedent(""" class {name}(int, Enum): {members} @@ -78,10 +76,10 @@ class {name}(int, Enum): autogen_context.imports.add('from indico.core.db.sqlalchemy import PyIntEnum') if self.exclude_values: return '{}({}, exclude_values={{{}}})'.format(type(self).__name__, name, ', '.join( - '{}.{}'.format(name, x.name) for x in sorted(self.exclude_values) + f'{name}.{x.name}' for x in sorted(self.exclude_values) )) else: - return '{}({})'.format(type(self).__name__, name) + return f'{type(self).__name__}({name})' def marshmallow_get_field_kwargs(self): return {'enum': self.enum} @@ -91,7 +89,8 @@ def marshmallow_get_field_kwargs(self): def _type_before_parent_attach(type_, col): @listens_for(col, 'after_parent_attach') def _col_after_parent_attach(col, table): - e = CheckConstraint(type_coerce(col, type_).in_(x.value for x in type_.enum if x not in type_.exclude_values), - 'valid_enum_{}'.format(col.name)) + int_col = type_coerce(col, SmallInteger) + e = CheckConstraint(int_col.in_(x.value for x in type_.enum if x not in type_.exclude_values), + f'valid_enum_{col.name}') e.info['alembic_dont_render'] = True assert e.table is table diff --git a/indico/core/db/sqlalchemy/custom/ip_network.py b/indico/core/db/sqlalchemy/custom/ip_network.py index f8c0ed17faa..4f3820d78e8 100644 --- a/indico/core/db/sqlalchemy/custom/ip_network.py +++ b/indico/core/db/sqlalchemy/custom/ip_network.py @@ -1,24 +1,22 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import ipaddress from sqlalchemy import TypeDecorator from sqlalchemy.dialects.postgresql import CIDR -class _AlwaysSortableMixin(object): +class _AlwaysSortableMixin: def __lt__(self, other): if self._version != other._version: return self._version < other._version else: - return super(_AlwaysSortableMixin, self).__lt__(other) + return super().__lt__(other) class IPv4Network(_AlwaysSortableMixin, ipaddress.IPv4Network): @@ -32,7 +30,7 @@ class IPv6Network(_AlwaysSortableMixin, ipaddress.IPv6Network): def _ip_network(address, strict=True): # based on `ipaddress.ip_network` but returns always-sortable classes # since sqlalchemy needs to be able to sort all values of a type - address = unicode(address) + address = str(address) try: return IPv4Network(address, strict) except (ipaddress.AddressValueError, ipaddress.NetmaskValueError): @@ -52,7 +50,7 @@ class PyIPNetwork(TypeDecorator): impl = CIDR def process_bind_param(self, value, dialect): - return unicode(_ip_network(value)) if value is not None else None + return str(_ip_network(value)) if value is not None else None def process_result_value(self, value, dialect): return _ip_network(value) if value is not None else None @@ -62,4 +60,4 @@ def coerce_set_value(self, value): def alembic_render_type(self, autogen_context, toplevel_code): autogen_context.imports.add('from indico.core.db.sqlalchemy import PyIPNetwork') - return '{}()'.format(type(self).__name__) + return f'{type(self).__name__}()' diff --git a/indico/core/db/sqlalchemy/custom/least.py b/indico/core/db/sqlalchemy/custom/least.py index c5baf4ec9d7..1fce773725f 100644 --- a/indico/core/db/sqlalchemy/custom/least.py +++ b/indico/core/db/sqlalchemy/custom/least.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/db/sqlalchemy/custom/natsort.py b/indico/core/db/sqlalchemy/custom/natsort.py index fe155bf4fbb..535555e28ca 100644 --- a/indico/core/db/sqlalchemy/custom/natsort.py +++ b/indico/core/db/sqlalchemy/custom/natsort.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy import DDL, text diff --git a/indico/core/db/sqlalchemy/custom/static_array.py b/indico/core/db/sqlalchemy/custom/static_array.py index 36063fcff74..96737540eae 100644 --- a/indico/core/db/sqlalchemy/custom/static_array.py +++ b/indico/core/db/sqlalchemy/custom/static_array.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -12,7 +12,7 @@ StaticArray class and functions that SQLAlchemy can process instead of non hashable lists """ -from cStringIO import StringIO +from io import StringIO from sqlalchemy import String, types from sqlalchemy.dialects.postgresql import ARRAY @@ -25,7 +25,7 @@ class StaticArray(types.TypeDecorator): impl = types.TypeEngine def __init__(self): - super(StaticArray, self).__init__() + super().__init__() self.__supported = {PGDialect: ARRAY} def load_dialect_impl(self, dialect): diff --git a/indico/core/db/sqlalchemy/custom/unaccent.py b/indico/core/db/sqlalchemy/custom/unaccent.py index 18e16e277b3..9749de295c1 100644 --- a/indico/core/db/sqlalchemy/custom/unaccent.py +++ b/indico/core/db/sqlalchemy/custom/unaccent.py @@ -1,19 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy import DDL, Index, text from sqlalchemy.event import listens_for from sqlalchemy.sql import func from sqlalchemy.sql.elements import conv -from indico.util.string import to_unicode - # if you wonder why search_path is set and the two-argument `unaccent` function is used, # see this post on stackoverflow: http://stackoverflow.com/a/11007216/298479 @@ -41,7 +37,7 @@ def create_unaccent_function(conn): def define_unaccented_lowercase_index(column): - """Defines an index that uses the indico_unaccent function. + """Define an index that uses the indico_unaccent function. Since this is usually used for searching, the column's value is also converted to lowercase before being unaccented. To make proper @@ -61,14 +57,14 @@ def _after_create(target, conn, **kw): col_func = func.indico.indico_unaccent(func.lower(column)) index_kwargs = {'postgresql_using': 'gin', 'postgresql_ops': {col_func.key: 'gin_trgm_ops'}} - Index(conv('ix_{}_{}_unaccent'.format(column.table.name, column.name)), col_func, **index_kwargs).create(conn) + Index(conv(f'ix_{column.table.name}_{column.name}_unaccent'), col_func, **index_kwargs).create(conn) def unaccent_match(column, value, exact): from indico.core.db import db - value = to_unicode(value).replace('%', r'\%').replace('_', r'\_').lower() + value = value.replace('%', r'\%').replace('_', r'\_').lower() if not exact: - value = '%{}%'.format(value) + value = f'%{value}%' # we always use LIKE, even for an exact match. when using the pg_trgm indexes this is # actually faster than `=` return db.func.indico.indico_unaccent(db.func.lower(column)).ilike(db.func.indico.indico_unaccent(value)) diff --git a/indico/core/db/sqlalchemy/custom/utcdatetime.py b/indico/core/db/sqlalchemy/custom/utcdatetime.py index 06a12848441..e72daf79197 100644 --- a/indico/core/db/sqlalchemy/custom/utcdatetime.py +++ b/indico/core/db/sqlalchemy/custom/utcdatetime.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/db/sqlalchemy/descriptions.py b/indico/core/db/sqlalchemy/descriptions.py index 50cb8684430..6de381254bd 100644 --- a/indico/core/db/sqlalchemy/descriptions.py +++ b/indico/core/db/sqlalchemy/descriptions.py @@ -1,19 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from sqlalchemy.ext.hybrid import hybrid_property from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum +from indico.util.enum import RichIntEnum from indico.util.string import MarkdownText, PlainText, RichMarkup -from indico.util.struct.enum import RichIntEnum class RenderMode(RichIntEnum): @@ -32,8 +30,8 @@ class RenderMode(RichIntEnum): } -class RenderModeMixin(object): - """Mixin to add a plaintext/html/markdown-enabled column.""" +class RenderModeMixin: + """Mixin to add a plaintext/html/markdown-enabled column.""" possible_render_modes = {RenderMode.plain_text} default_render_mode = RenderMode.plain_text diff --git a/indico/core/db/sqlalchemy/links.py b/indico/core/db/sqlalchemy/links.py index 0b81a6366c5..76eb372b348 100644 --- a/indico/core/db/sqlalchemy/links.py +++ b/indico/core/db/sqlalchemy/links.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import partial from itertools import chain @@ -17,8 +15,8 @@ from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum from indico.util.decorators import strict_classproperty +from indico.util.enum import RichIntEnum from indico.util.i18n import _ -from indico.util.struct.enum import RichIntEnum class LinkType(RichIntEnum): @@ -46,28 +44,28 @@ class LinkType(RichIntEnum): def _make_checks(allowed_link_types): - available_columns = set(chain.from_iterable(cols for type_, cols in _columns_for_types.iteritems() + available_columns = set(chain.from_iterable(cols for type_, cols in _columns_for_types.items() if type_ in allowed_link_types)) - yield db.CheckConstraint('(event_id IS NULL) = (link_type = {})'.format(LinkType.category), 'valid_event_id') + yield db.CheckConstraint(f'(event_id IS NULL) = (link_type = {LinkType.category})', 'valid_event_id') for link_type in allowed_link_types: required_cols = available_columns & _columns_for_types[link_type] forbidden_cols = available_columns - required_cols - criteria = ['{} IS NULL'.format(col) for col in sorted(forbidden_cols)] - criteria += ['{} IS NOT NULL'.format(col) for col in sorted(required_cols)] + criteria = [f'{col} IS NULL' for col in sorted(forbidden_cols)] + criteria += [f'{col} IS NOT NULL' for col in sorted(required_cols)] condition = 'link_type != {} OR ({})'.format(link_type, ' AND '.join(criteria)) - yield db.CheckConstraint(condition, 'valid_{}_link'.format(link_type.name)) + yield db.CheckConstraint(condition, f'valid_{link_type.name}_link') def _make_uniques(allowed_link_types, extra_criteria=None): for link_type in allowed_link_types: - where = ['link_type = {}'.format(link_type.value)] + where = [f'link_type = {link_type.value}'] if extra_criteria is not None: where += list(extra_criteria) yield db.Index(None, *_columns_for_types[link_type], unique=True, postgresql_where=db.text(' AND '.join(where))) -class LinkMixin(object): +class LinkMixin: #: The link types that are supported. Can be overridden in the #: model using the mixin. Affects the table structure, so any #: changes to it should go along with a migration step! @@ -89,13 +87,13 @@ class LinkMixin(object): def __auto_table_args(cls): args = tuple(_make_checks(cls.allowed_link_types)) if cls.unique_links: - extra_criteria = [cls.unique_links] if isinstance(cls.unique_links, basestring) else None + extra_criteria = [cls.unique_links] if isinstance(cls.unique_links, str) else None args = args + tuple(_make_uniques(cls.allowed_link_types, extra_criteria)) return args @classmethod def register_link_events(cls): - """Registers sqlalchemy events needed by this mixin. + """Register sqlalchemy events needed by this mixin. Call this method after the definition of a model which uses this mixin class. @@ -123,11 +121,11 @@ def _set_event_obj(fn, target, value, *unused): assert event is not None target.event = event - for rel, fn in event_mapping.iteritems(): + for rel, fn in event_mapping.items(): if rel is not None: listen(rel, 'set', partial(_set_event_obj, fn)) - for rel, link_type in type_mapping.iteritems(): + for rel, link_type in type_mapping.items(): if rel is not None: listen(rel, 'set', partial(_set_link_type, link_type)) @@ -337,7 +335,7 @@ def object(self, obj): elif isinstance(obj, db.m.SubContribution): self.subcontribution = obj else: - raise TypeError('Unexpected object: {}'.format(obj)) + raise TypeError(f'Unexpected object: {obj}') @object.comparator def object(cls): @@ -345,15 +343,15 @@ def object(cls): @property def link_repr(self): - """A kwargs-style string suitable for the object's repr""" + """A kwargs-style string suitable for the object's repr.""" info = [('link_type', self.link_type.name if self.link_type is not None else 'None')] info.extend((key, getattr(self, key)) for key in _all_columns if getattr(self, key) is not None) - return ', '.join('{}={}'.format(key, value) for key, value in info) + return ', '.join(f'{key}={value}' for key, value in info) @property def link_event_log_data(self): """ - Returns a dict containing information about the linked object + Return a dict containing information about the linked object suitable for the event log. It does not return any information for an object linked to a @@ -400,4 +398,4 @@ def __eq__(self, other): return db.and_(self.cls.link_type == LinkType.subcontribution, self.cls.subcontribution_id == other.id) else: - raise ValueError('Unexpected object type {}: {}'.format(type(other), other)) + raise ValueError(f'Unexpected object type {type(other)}: {other}') diff --git a/indico/core/db/sqlalchemy/locations.py b/indico/core/db/sqlalchemy/locations.py index 0209d680a6e..84232ad6b5a 100644 --- a/indico/core/db/sqlalchemy/locations.py +++ b/indico/core/db/sqlalchemy/locations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.event import listens_for from sqlalchemy.ext.declarative import declared_attr @@ -14,7 +12,7 @@ from indico.util.decorators import strict_classproperty -class LocationMixin(object): +class LocationMixin: """Mixin to store location information in a model. A location in this context can be either a reference to a room in @@ -51,7 +49,7 @@ def __auto_table_args(cls): @classmethod def register_location_events(cls): - """Registers sqlalchemy events needed by this mixin. + """Register sqlalchemy events needed by this mixin. Call this method after the definition of a model which uses this mixin class. @@ -199,20 +197,30 @@ def venue_name(self): def venue_name(self, venue_name): self.own_venue_name = venue_name - def get_room_name(self, full=True): + def get_room_name(self, full=True, verbose=False): """The name of the room where this item is located. - :param full: by default, if the room has a "friendly name" too - (e.g. 'Main Amphitheatre'), a composite name will be - returned. If ``full`` is set to ``False``, only the - "friendly name" will be returned in that case. + If both ``full`` and ``verbose`` are set to ``False``, the + "friendly name" will be returned in that case. Both ``full`` and + ``verbose`` cannot be set to ``True``. + + :param full: If the room has a "friendly name" (e.g. 'Main + Amphitheatre'), a composite name will be returned. + :param verbose: The `verbose_name` of the room will be returned. """ + assert sum([full, verbose]) <= 1 if self.inherit_location and self.location_parent is None: return '' room = self.room if room is not None: - return room.full_name if full else room.name - return self.own_room_name if not self.inherit_location else self.location_parent.room_name + if full: + return room.full_name + elif verbose and room.verbose_name: + return room.verbose_name + else: + return room.name + return (self.own_room_name if not self.inherit_location + else self.location_parent.get_room_name(full=full, verbose=verbose)) @property def room_name(self): @@ -225,7 +233,7 @@ def room_name(self, room_name): @property def has_location_info(self): - """Whether the object has basic location information set""" + """Whether the object has basic location information set.""" return bool(self.venue_name or self.room_name) @property diff --git a/indico/core/db/sqlalchemy/logging.py b/indico/core/db/sqlalchemy/logging.py index 3dc2bb79933..99650dccd39 100644 --- a/indico/core/db/sqlalchemy/logging.py +++ b/indico/core/db/sqlalchemy/logging.py @@ -1,16 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import, unicode_literals - import logging import pprint import time import traceback +from collections.abc import Mapping from flask import appcontext_tearing_down, current_app, g, has_request_context, request, request_tearing_down from sqlalchemy.engine import Engine @@ -34,19 +33,23 @@ def _interesting_tb_item(item, paths): return (item[0].endswith('.tpl.py') or any(item[0].startswith(p) for p in paths)) and 'sqlalchemy' not in item[0] +def _frame_to_tuple(frame): + return frame.filename, frame.lineno, frame.name, frame.line + + def _get_sql_line(): - paths = [current_app.root_path] + [p.root_path for p in plugin_engine.get_active_plugins().itervalues()] + paths = [current_app.root_path] + [p.root_path for p in plugin_engine.get_active_plugins().values()] stack = [item for item in reversed(traceback.extract_stack()) if _interesting_tb_item(item, paths)] for i, item in enumerate(stack): return {'file': item[0], 'line': item[1], 'function': item[2], - 'items': stack[i:i+5]} + 'items': [_frame_to_tuple(frame) for frame in stack[i:i+5]]} def _fix_param(param): - if hasattr(param, 'iteritems'): - return {k: _fix_param(v) for k, v in param.iteritems()} + if isinstance(param, Mapping): + return {k: _fix_param(v) for k, v in param.items()} return '' if param.__class__.__name__ == 'Binary' else param @@ -79,13 +82,13 @@ def before_cursor_execute(conn, cursor, statement, parameters, context, executem ).rstrip() else: # UPDATEs can't be traced back to their source since they are executed only on flush - log_msg = 'Start Query:\n{0}\n{1}'.format( + log_msg = 'Start Query:\n{}\n{}'.format( _prettify_sql(statement), _prettify_params(parameters) if parameters else '' ).rstrip() # psycopg2._psycopg.Binary objects are extremely weird and don't work in isinstance checks - if hasattr(parameters, 'iteritems'): - parameters = {k: _fix_param(v) for k, v in parameters.iteritems()} + if hasattr(parameters, 'items'): + parameters = {k: _fix_param(v) for k, v in parameters.items()} else: parameters = tuple(_fix_param(v) for v in parameters) logger.debug(log_msg, diff --git a/indico/core/db/sqlalchemy/migration.py b/indico/core/db/sqlalchemy/migration.py index 43324d27c5f..25685f91ca0 100644 --- a/indico/core/db/sqlalchemy/migration.py +++ b/indico/core/db/sqlalchemy/migration.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - import os import alembic.command @@ -35,14 +33,14 @@ class PluginScriptDirectory(ScriptDirectory): versions = None def __init__(self, *args, **kwargs): - super(PluginScriptDirectory, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.dir = PluginScriptDirectory.dir # use __dict__ since it's a memoized property self.__dict__['_version_locations'] = [current_plugin.alembic_versions_path] @classmethod def from_config(cls, config): - instance = super(PluginScriptDirectory, cls).from_config(config) + instance = super().from_config(config) instance.dir = PluginScriptDirectory.dir instance.__dict__['_version_locations'] = [current_plugin.alembic_versions_path] return instance @@ -62,6 +60,9 @@ def _require_extensions(*names): def _require_pg_version(version): # convert version string such as '9.4.10' to `90410` which is the # format used by server_version_num + # FIXME: this will not work for versions >= 10, since the second segment there is the patch version + # but once we require a newer postgres, we can ditch the logic here and only use the major version + # since that will be the only relevant version number req_version = sum(segment * 10**(4 - 2*i) for i, segment in enumerate(map(int, version.split('.')))) cur_version = db.engine.execute("SELECT current_setting('server_version_num')::int").scalar() if cur_version >= req_version: @@ -97,7 +98,7 @@ def prepare_db(empty=False, root_path=None, verbose=True): alembic.command.ScriptDirectory = PluginScriptDirectory plugin_msg = cformat("%{cyan}Setting the alembic version of the %{cyan!}{}%{reset}%{cyan} " "plugin to HEAD%{reset}") - for plugin in plugin_engine.get_active_plugins().itervalues(): + for plugin in plugin_engine.get_active_plugins().values(): if not os.path.exists(plugin.alembic_versions_path): continue if verbose: @@ -108,7 +109,7 @@ def prepare_db(empty=False, root_path=None, verbose=True): tables = get_all_tables(db) tables['public'] = [t for t in tables['public'] if not t.startswith('alembic_version')] - if any(tables.viewvalues()): + if any(tables.values()): if verbose: print(cformat('%{red}Your database is not empty!')) print(cformat('%{yellow}If you just added a new table/model, create an alembic revision instead!')) diff --git a/indico/core/db/sqlalchemy/notes.py b/indico/core/db/sqlalchemy/notes.py index b30d25679c7..861464f6fa6 100644 --- a/indico/core/db/sqlalchemy/notes.py +++ b/indico/core/db/sqlalchemy/notes.py @@ -1,20 +1,20 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.notes.models.notes import EventNote from indico.modules.events.notes.util import can_edit_note from indico.util.caching import memoize_request -class AttachedNotesMixin(object): - """Allows for easy retrieval of structured information about - items attached to the object""" +class AttachedNotesMixin: + """ + Allow for easy retrieval of structured information about + items attached to the object. + """ # When set to ``True`` .has_note preload all notes that exist for the same event # Should be set to False when not applicable (no object.event property) PRELOAD_EVENT_NOTES = False diff --git a/indico/core/db/sqlalchemy/principals.py b/indico/core/db/sqlalchemy/principals.py index 2fbfbbc44a9..980196abada 100644 --- a/indico/core/db/sqlalchemy/principals.py +++ b/indico/core/db/sqlalchemy/principals.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import ARRAY from sqlalchemy.ext.declarative import declared_attr from sqlalchemy.ext.hybrid import Comparator, hybrid_method, hybrid_property @@ -16,9 +14,8 @@ from indico.core.db.sqlalchemy.util.models import get_simple_column_attrs from indico.core.permissions import get_available_permissions from indico.util.decorators import classproperty, strict_classproperty -from indico.util.fossilize import Fossilizable, IFossil, fossilizes -from indico.util.string import format_repr, return_ascii -from indico.util.struct.enum import IndicoEnum +from indico.util.enum import IndicoEnum +from indico.util.string import format_repr class PrincipalType(int, IndicoEnum): @@ -47,14 +44,14 @@ def _make_check(type_, allow_emails, allow_networks, allow_event_roles, allow_ca all_cols.add('registration_form_id') required_cols = all_cols & set(cols) forbidden_cols = all_cols - required_cols - criteria = ['{} IS NULL'.format(col) for col in sorted(forbidden_cols)] - criteria += ['{} IS NOT NULL'.format(col) for col in sorted(required_cols)] + criteria = [f'{col} IS NULL' for col in sorted(forbidden_cols)] + criteria += [f'{col} IS NOT NULL' for col in sorted(required_cols)] condition = 'type != {} OR ({})'.format(type_, ' AND '.join(criteria)) - return db.CheckConstraint(condition, 'valid_{}'.format(type_.name)) + return db.CheckConstraint(condition, f'valid_{type_.name}') def serialize_email_principal(email): - """Serialize email principal to a simple dict""" + """Serialize email principal to a simple dict.""" return { '_type': 'Email', 'email': email.email, @@ -64,39 +61,14 @@ def serialize_email_principal(email): } -class IEmailPrincipalFossil(IFossil): - def getId(self): - pass - getId.produce = lambda x: x.email - - def getIdentifier(self): - pass - getIdentifier.produce = lambda x: 'Email:{}'.format(x.email) - - def getEmail(self): - pass - getEmail.produce = lambda x: x.email - - def getName(self): - pass - getName.produce = lambda x: x.name - - -class EmailPrincipal(Fossilizable): - """Wrapper for email principals +class EmailPrincipal: + """Wrapper for email principals. :param email: The email address. """ principal_type = PrincipalType.email - is_network = False - is_group = False - is_single_person = True - is_event_role = False - is_category_role = False - is_registration_form = False principal_order = 0 - fossilizes(IEmailPrincipalFossil) def __init__(self, email): self.email = email.lower() @@ -105,10 +77,6 @@ def __init__(self, email): def name(self): return self.email - @property - def as_legacy(self): - return self - @property def user(self): from indico.modules.users import User @@ -116,7 +84,7 @@ def user(self): @property def identifier(self): - return 'Email:{}'.format(self.email) + return f'Email:{self.email}' def __eq__(self, other): return isinstance(other, EmailPrincipal) and self.email == other.email @@ -132,12 +100,11 @@ def __contains__(self, user): return False return self.email in user.all_emails - @return_ascii def __repr__(self): return format_repr(self, 'email') -class PrincipalMixin(object): +class PrincipalMixin: #: The name of the backref added to `User` and `LocalGroup`. #: For consistency, it is recommended to name the backref #: ``in_foo_acl`` with *foo* describing the ACL where this @@ -166,16 +133,16 @@ class PrincipalMixin(object): def __auto_table_args(cls): uniques = () if cls.unique_columns: - uniques = [db.Index('ix_uq_{}_user'.format(cls.__tablename__), 'user_id', *cls.unique_columns, unique=True, - postgresql_where=db.text('type = {}'.format(PrincipalType.user))), - db.Index('ix_uq_{}_local_group'.format(cls.__tablename__), 'local_group_id', *cls.unique_columns, - unique=True, postgresql_where=db.text('type = {}'.format(PrincipalType.local_group))), - db.Index('ix_uq_{}_mp_group'.format(cls.__tablename__), 'mp_group_provider', 'mp_group_name', + uniques = [db.Index(f'ix_uq_{cls.__tablename__}_user', 'user_id', *cls.unique_columns, unique=True, + postgresql_where=db.text(f'type = {PrincipalType.user}')), + db.Index(f'ix_uq_{cls.__tablename__}_local_group', 'local_group_id', *cls.unique_columns, + unique=True, postgresql_where=db.text(f'type = {PrincipalType.local_group}')), + db.Index(f'ix_uq_{cls.__tablename__}_mp_group', 'mp_group_provider', 'mp_group_name', *cls.unique_columns, unique=True, - postgresql_where=db.text('type = {}'.format(PrincipalType.multipass_group)))] + postgresql_where=db.text(f'type = {PrincipalType.multipass_group}'))] if cls.allow_emails: - uniques.append(db.Index('ix_uq_{}_email'.format(cls.__tablename__), 'email', *cls.unique_columns, - unique=True, postgresql_where=db.text('type = {}'.format(PrincipalType.email)))) + uniques.append(db.Index(f'ix_uq_{cls.__tablename__}_email', 'email', *cls.unique_columns, + unique=True, postgresql_where=db.text(f'type = {PrincipalType.email}'))) indexes = [db.Index(None, 'mp_group_provider', 'mp_group_name')] checks = [_make_check(PrincipalType.user, cls.allow_emails, cls.allow_networks, cls.allow_event_roles, cls.allow_category_roles, cls.allow_registration_forms, 'user_id'), @@ -452,7 +419,7 @@ def principal(self, value): elif self.type == PrincipalType.user: self.user = value else: - raise ValueError('Unexpected principal type: {}'.format(self.type)) + raise ValueError(f'Unexpected principal type: {self.type}') @principal.comparator def principal(cls): @@ -488,7 +455,7 @@ def get_users(self): return set() def merge_privs(self, other): - """Merges the privileges of another principal + """Merge the privileges of another principal. :param other: Another principal object. """ @@ -499,7 +466,7 @@ def current_data(self): @classmethod def merge_users(cls, target, source, relationship_attr): - """Merges two users in the ACL. + """Merge two users in the ACL. :param target: The target user of the merge. :param source: The user that is being merged into `target`. @@ -523,8 +490,9 @@ def merge_users(cls, target, source, relationship_attr): @classmethod def replace_email_with_user(cls, user, relationship_attr): """ - Replaces all email-based entries matching the user's email + Replace all email-based entries matching the user's email addresses with user-based entries. + If the user is already in the ACL, the two entries are merged. :param user: A User object. @@ -536,8 +504,8 @@ def replace_email_with_user(cls, user, relationship_attr): """ assert cls.allow_emails updated = set() - query = (cls - .find(cls.email.in_(user.all_emails)) + query = (cls.query + .filter(cls.email.in_(user.all_emails)) .options(noload('user'), noload('local_group'), joinedload(relationship_attr).load_only('id'))) for entry in query: parent = getattr(entry, relationship_attr) @@ -599,14 +567,14 @@ def __auto_table_args(cls): @classproperty @classmethod def principal_for_obj(cls): - if isinstance(cls.principal_for, basestring): - return db.Model._decl_class_registry[cls.principal_for] + if isinstance(cls.principal_for, str): + return db.Model.registry._class_registry[cls.principal_for] else: return cls.principal_for @hybrid_method def has_management_permission(self, permission=None, explicit=False): - """Checks whether a principal has a certain management permission. + """Check whether a principal has a certain management permission. The check always succeeds if the user is a full manager; in that case the list of permissions is ignored. @@ -622,7 +590,7 @@ def has_management_permission(self, permission=None, explicit=False): return self.full_access elif not explicit and self.full_access: return True - valid_permissions = get_available_permissions(self.principal_for_obj).viewkeys() + valid_permissions = get_available_permissions(self.principal_for_obj).keys() current_permissions = set(self.permissions) & valid_permissions if permission == 'ANY': return bool(current_permissions) @@ -636,12 +604,12 @@ def has_management_permission(cls, permission=None, explicit=False): if explicit: raise ValueError('permission must be specified if explicit=True') return cls.full_access - valid_permissions = get_available_permissions(cls.principal_for_obj).viewkeys() + valid_permissions = get_available_permissions(cls.principal_for_obj).keys() if permission == 'ANY': crit = (cls.permissions.op('&&')(db.func.cast(valid_permissions, ARRAY(db.String)))) else: assert permission in valid_permissions, \ - "invalid permission '{}' for object '{}'".format(permission, cls.principal_for_obj) + f"invalid permission '{permission}' for object '{cls.principal_for_obj}'" crit = (cls.permissions.op('&&')(db.func.cast([permission], ARRAY(db.String)))) if explicit: return crit @@ -687,7 +655,7 @@ def __eq__(self, other): elif other.principal_type == PrincipalType.user: criteria = [self.cls.user_id == other.id] else: - raise ValueError('Unexpected object type {}: {}'.format(type(other), other)) + raise ValueError(f'Unexpected object type {type(other)}: {other}') return db.and_(self.cls.type == other.principal_type, *criteria) diff --git a/indico/core/db/sqlalchemy/protection.py b/indico/core/db/sqlalchemy/protection.py index 2af48e0dbea..f390b4f1578 100644 --- a/indico/core/db/sqlalchemy/protection.py +++ b/indico/core/db/sqlalchemy/protection.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import itertools from flask import has_request_context, session @@ -22,9 +20,9 @@ from indico.core.db.sqlalchemy.principals import EmailPrincipal, PrincipalType from indico.core.permissions import get_available_permissions from indico.util.caching import memoize_request +from indico.util.enum import RichIntEnum from indico.util.i18n import _ from indico.util.signals import values_from_signal -from indico.util.struct.enum import RichIntEnum from indico.util.user import iter_acl from indico.web.util import jsonify_template @@ -36,7 +34,7 @@ class ProtectionMode(RichIntEnum): protected = 2 -class ProtectionMixin(object): +class ProtectionMixin: #: Whether the object protection mode is disabled disable_protection_mode = False #: The protection modes that are not allowed. Can be overridden @@ -60,7 +58,7 @@ class ProtectionMixin(object): @classmethod def register_protection_events(cls): - """Registers sqlalchemy events needed by this mixin. + """Register sqlalchemy events needed by this mixin. Call this method after the definition of a model which uses this mixin class. @@ -120,7 +118,7 @@ def is_inheriting(self): @hybrid_property def is_self_protected(self): - """Checks whether the object itself is protected. + """Check whether the object itself is protected. If you also care about inherited protection from a parent, use `is_protected` instead. @@ -132,7 +130,7 @@ def is_self_protected(self): @property def is_protected(self): """ - Checks whether ths object is protected, either by itself or + Check whether this object is protected, either by itself or by inheriting from a protected object. """ if self.disable_protection_mode: @@ -147,11 +145,11 @@ def protection_repr(self): if self.disable_protection_mode: return 'protection_mode=disabled' protection_mode = self.protection_mode.name if self.protection_mode is not None else None - return 'protection_mode={}'.format(protection_mode) + return f'protection_mode={protection_mode}' @property def protection_parent(self): - """The parent object to consult for ProtectionMode.inheriting""" + """The parent object to consult for ProtectionMode.inheriting.""" raise NotImplementedError def _check_can_access_override(self, user, allow_admin, authorized=None): @@ -169,7 +167,7 @@ def is_user_admin(user): @memoize_request def can_access(self, user, allow_admin=True): - """Checks if the user can access the object. + """Check if the user can access the object. :param user: The :class:`.User` to check. May be None if the user is not logged in. @@ -217,7 +215,7 @@ def can_access(self, user, allow_admin=True): # This should be the case for the top-level object, # i.e. the root category, which shouldn't allow # ProtectionMode.inheriting as it makes no sense. - raise TypeError('protection_parent of {} is None'.format(self)) + raise TypeError(f'protection_parent of {self} is None') elif hasattr(parent, 'can_access'): rv = parent.can_access(user, allow_admin=allow_admin) else: @@ -226,7 +224,7 @@ def can_access(self, user, allow_admin=True): else: # should never happen, but since this is a sensitive area # we better fail loudly if we have garbage - raise ValueError('Invalid protection mode: {}'.format(self.protection_mode)) + raise ValueError(f'Invalid protection mode: {self.protection_mode}') override = self._check_can_access_override(user, allow_admin=allow_admin, authorized=rv) return override if override is not None else rv @@ -261,10 +259,10 @@ def set_session_access_key(self, access_key): @property def _access_key_session_key(self): cls, pks = inspect(self).identity_key[:2] - return '{}-{}'.format(cls.__name__, '-'.join(map(unicode, pks))) + return '{}-{}'.format(cls.__name__, '-'.join(map(str, pks))) def update_principal(self, principal, read_access=None, quiet=False): - """Updates access privileges for the given principal. + """Update access privileges for the given principal. :param principal: A `User`, `GroupProxy` or `EmailPrincipal` instance. :param read_access: If the principal should have explicit read @@ -300,7 +298,7 @@ def update_principal(self, principal, read_access=None, quiet=False): return entry def remove_principal(self, principal, quiet=False): - """Revokes all access privileges for the given principal. + """Revoke all access privileges for the given principal. This method doesn't do anything if the user is not in the object's ACL. @@ -327,7 +325,7 @@ def get_inherited_acl(self): class ProtectionManagersMixin(ProtectionMixin): @property def all_manager_emails(self): - """Return the emails of all managers""" + """Return the emails of all managers.""" # We ignore email principals here. They never signed up in indico anyway... return {p.principal.email for p in self.acl_entries @@ -335,7 +333,7 @@ def all_manager_emails(self): @memoize_request def can_manage(self, user, permission=None, allow_admin=True, check_parent=True, explicit_permission=False): - """Checks if the user can manage the object. + """Check if the user can manage the object. :param user: The :class:`.User` to check. May be None if the user is not logged in. @@ -363,13 +361,16 @@ def can_manage(self, user, permission=None, allow_admin=True, check_parent=True, to ``False``. """ if permission is not None and permission != 'ANY' and permission not in get_available_permissions(type(self)): - raise ValueError("permission '{}' is not valid for '{}' objects".format(permission, type(self).__name__)) + raise ValueError(f"permission '{permission}' is not valid for '{type(self).__name__}' objects") if user is None: # An unauthorized user is never allowed to perform management operations. # Not even signals may override this since management code generally # expects session.user to be not None. return False + if user.is_system: + # A system user has no email and thus access checks (against groups) may fail + return False # Trigger signals for protection overrides rv = values_from_signal(signals.acl.can_manage.send(type(self), obj=self, user=user, permission=permission, @@ -404,11 +405,11 @@ def can_manage(self, user, permission=None, allow_admin=True, check_parent=True, elif hasattr(parent, 'can_manage'): return parent.can_manage(user, allow_admin=allow_admin) else: - raise TypeError('protection_parent of {} is of invalid type {} ({})'.format(self, type(parent), parent)) + raise TypeError(f'protection_parent of {self} is of invalid type {type(parent)} ({parent})') def update_principal(self, principal, read_access=None, full_access=None, permissions=None, add_permissions=None, del_permissions=None, quiet=False): - """Updates access privileges for the given principal. + """Update access privileges for the given principal. If the principal is not in the ACL, it will be added if necessary. If the changes remove all its privileges, it @@ -455,7 +456,7 @@ def update_principal(self, principal, read_access=None, full_access=None, permis new_permissions |= add_permissions if del_permissions: new_permissions -= del_permissions - invalid_permissions = new_permissions - get_available_permissions(type(self)).viewkeys() + invalid_permissions = new_permissions - get_available_permissions(type(self)).keys() if invalid_permissions: raise ValueError('Invalid permissions: {}'.format(', '.join(invalid_permissions))) entry.permissions = sorted(new_permissions) @@ -515,7 +516,7 @@ def get_access_list(self, skip_managers=False, skip_self_acl=False): def _get_acl_data(obj, principal): - """Helper function to get the necessary data for ACL modifications + """Helper function to get the necessary data for ACL modifications. :param obj: A `ProtectionMixin` instance :param principal: A User or GroupProxy uinstance @@ -528,7 +529,7 @@ def _get_acl_data(obj, principal): def _resolve_principal(principal): - """Helper function to convert an email principal to a user if possible + """Helper function to convert an email principal to a user if possible. :param principal: A `User`, `GroupProxy` or `EmailPrincipal` instance. """ diff --git a/indico/core/db/sqlalchemy/review_comments.py b/indico/core/db/sqlalchemy/review_comments.py index a4d99ec0460..8ce88fe8193 100644 --- a/indico/core/db/sqlalchemy/review_comments.py +++ b/indico/core/db/sqlalchemy/review_comments.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db diff --git a/indico/core/db/sqlalchemy/review_questions.py b/indico/core/db/sqlalchemy/review_questions.py index 936ffb12ec6..d1b9d9ea023 100644 --- a/indico/core/db/sqlalchemy/review_questions.py +++ b/indico/core/db/sqlalchemy/review_questions.py @@ -1,29 +1,27 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.ext.declarative import declared_attr from sqlalchemy.ext.hybrid import hybrid_property from indico.core.db import db -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr def _get_next_position(cls): def __get_next_position(context): event_id = context.current_parameters['event_id'] - res = db.session.query(db.func.max(cls.position)).filter_by(event_id=event_id, is_deleted=False).one() + res = db.session.query(db.func.max(cls.position)).filter(cls.event_id == event_id, ~cls.is_deleted).one() return (res[0] or 0) + 1 return __get_next_position -class ReviewQuestionMixin(object): +class ReviewQuestionMixin: #: name of backref from event to questions event_backref_name = None @@ -120,7 +118,6 @@ def event(cls): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'event_id', no_score=False, is_deleted=False, _text=self.title) diff --git a/indico/core/db/sqlalchemy/review_ratings.py b/indico/core/db/sqlalchemy/review_ratings.py index 63a3e92cfb6..1c0b79778d6 100644 --- a/indico/core/db/sqlalchemy/review_ratings.py +++ b/indico/core/db/sqlalchemy/review_ratings.py @@ -1,20 +1,18 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr -class ReviewRatingMixin(object): +class ReviewRatingMixin: question_class = None review_class = None @@ -29,7 +27,7 @@ def id(cls): def question_id(cls): return db.Column( db.Integer, - db.ForeignKey('{}.id'.format(cls.question_class.__table__.fullname)), + db.ForeignKey(f'{cls.question_class.__table__.fullname}.id'), index=True, nullable=False ) @@ -38,7 +36,7 @@ def question_id(cls): def review_id(cls): return db.Column( db.Integer, - db.ForeignKey('{}.id'.format(cls.review_class.__table__.fullname)), + db.ForeignKey(f'{cls.review_class.__table__.fullname}.id'), index=True, nullable=False ) @@ -74,6 +72,5 @@ def review(cls): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'review_id', 'question_id') diff --git a/indico/core/db/sqlalchemy/searchable_titles.py b/indico/core/db/sqlalchemy/searchable_titles.py index 10e52bd8a0f..a10ab3f225a 100644 --- a/indico/core/db/sqlalchemy/searchable_titles.py +++ b/indico/core/db/sqlalchemy/searchable_titles.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db @@ -14,7 +12,7 @@ from indico.util.decorators import strict_classproperty -class SearchableTitleMixin(object): +class SearchableTitleMixin: """Mixin to add a fulltext-searchable title column.""" #: Whether the title column may not be empty @@ -24,7 +22,7 @@ class SearchableTitleMixin(object): @classmethod def __auto_table_args(cls): args = [ - db.Index('ix_{}_title_fts'.format(cls.__tablename__), db.func.to_tsvector('simple', cls.title), + db.Index(f'ix_{cls.__tablename__}_title_fts', db.func.to_tsvector('simple', cls.title), postgresql_using='gin') ] if cls.title_required: @@ -50,5 +48,5 @@ def title_matches(cls, search_string, exact=False): crit = db.func.to_tsvector('simple', cls.title).match(preprocess_ts_string(search_string), postgresql_regconfig='simple') if exact: - crit = crit & cls.title.ilike('%{}%'.format(escape_like(search_string))) + crit = crit & cls.title.ilike(f'%{escape_like(search_string)}%') return crit diff --git a/indico/core/db/sqlalchemy/util/management.py b/indico/core/db/sqlalchemy/util/management.py index 00f97529217..969d09be3ad 100644 --- a/indico/core/db/sqlalchemy/util/management.py +++ b/indico/core/db/sqlalchemy/util/management.py @@ -1,14 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import print_function, unicode_literals - -from sqlalchemy import ForeignKeyConstraint, MetaData, Table -from sqlalchemy.engine.reflection import Inspector +from sqlalchemy import ForeignKeyConstraint, MetaData, Table, inspect from sqlalchemy.sql.ddl import DropConstraint, DropSchema, DropTable from indico.core.db.sqlalchemy.protection import ProtectionMode @@ -64,23 +61,23 @@ def get_all_tables(db): - """Returns a dict containing all tables grouped by schema""" - inspector = Inspector.from_engine(db.engine) + """Return a dict containing all tables grouped by schema.""" + inspector = inspect(db.engine) schemas = sorted(set(inspector.get_schema_names()) - {'information_schema'}) return dict(zip(schemas, (inspector.get_table_names(schema=schema) for schema in schemas))) def delete_all_tables(db): - """Drops all tables in the database""" + """Drop all tables in the database.""" conn = db.engine.connect() transaction = conn.begin() - inspector = Inspector.from_engine(db.engine) + inspector = inspect(db.engine) metadata = MetaData() all_schema_tables = get_all_tables(db) tables = [] all_fkeys = [] - for schema, schema_tables in all_schema_tables.iteritems(): + for schema, schema_tables in all_schema_tables.items(): for table_name in schema_tables: fkeys = [ForeignKeyConstraint((), (), name=fk['name']) for fk in inspector.get_foreign_keys(table_name, schema=schema) @@ -106,11 +103,11 @@ def delete_all_tables(db): def create_all_tables(db, verbose=False, add_initial_data=True): - """Create all tables and required initial objects""" + """Create all tables and required initial objects.""" + from indico.core.oauth.models.applications import OAuthApplication, SystemAppType from indico.modules.categories import Category from indico.modules.designer import TemplateType from indico.modules.designer.models.templates import DesignerTemplate - from indico.modules.oauth.models.applications import OAuthApplication, SystemAppType from indico.modules.users import User if verbose: print(cformat('%{green}Creating tables')) diff --git a/indico/core/db/sqlalchemy/util/models.py b/indico/core/db/sqlalchemy/util/models.py index d554f0994fd..d75d1de54b6 100644 --- a/indico/core/db/sqlalchemy/util/models.py +++ b/indico/core/db/sqlalchemy/util/models.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os from copy import copy from importlib import import_module @@ -16,7 +14,7 @@ from sqlalchemy import Column, inspect, orm from sqlalchemy.event import listen, listens_for from sqlalchemy.ext.hybrid import hybrid_property -from sqlalchemy.orm import contains_eager, joinedload +from sqlalchemy.orm import joinedload from sqlalchemy.orm.attributes import get_history, set_committed_value from sqlalchemy.orm.exc import NoResultFound @@ -60,7 +58,7 @@ def has_rows(self): class IndicoModel(Model): - """Indico DB model""" + """Indico DB model.""" #: Whether relationship preloading is allowed. If disabled, #: the on-load event that populates relationship from the preload @@ -68,37 +66,6 @@ class IndicoModel(Model): allow_relationship_preloading = False query_class = IndicoBaseQuery - @classmethod - def find(cls, *args, **kwargs): - special_field_names = ('join', 'eager') - special_fields = {} - for key in special_field_names: - value = kwargs.pop('_{}'.format(key), ()) - if not isinstance(value, (list, tuple)): - value = (value,) - special_fields[key] = value - joined_eager = set(special_fields['eager']) & set(special_fields['join']) - options = [] - options += [joinedload(rel) for rel in special_fields['eager'] if rel not in joined_eager] - options += [contains_eager(rel) for rel in joined_eager] - return (cls.query - .filter_by(**kwargs) - .join(*special_fields['join']) - .filter(*args) - .options(*options)) - - @classmethod - def find_all(cls, *args, **kwargs): - return cls.find(*args, **kwargs).all() - - @classmethod - def find_first(cls, *args, **kwargs): - return cls.find(*args, **kwargs).first() - - @classmethod - def find_one(cls, *args, **kwargs): - return cls.find(*args, **kwargs).one() - @classmethod def get(cls, oid, is_deleted=None): """Get an object based on its primary key. @@ -164,12 +131,12 @@ def _populate_preloaded_relationships(cls, target, *unused): cache = g.get('relationship_cache', {}).get(type(target)) if not cache: return - for rel, value in cache['data'].get(target, {}).iteritems(): + for rel, value in cache['data'].get(target, {}).items(): if rel not in target.__dict__: set_committed_value(target, rel, value) def assign_id(self): - """Immediately assigns an ID to the object. + """Immediately assign an ID to the object. This only works if the table has exactly one serial column. It also "wastes" the ID if the new object is not actually @@ -196,7 +163,7 @@ def assign_id(self): setattr(self, attr_name, id_) def populate_from_dict(self, data, keys=None, skip=None, track_changes=True): - """Populates the object with values in a dictionary + """Populate the object with values in a dictionary. :param data: a dict containing values to populate the object. :param keys: If set, only keys from that list are populated. @@ -207,13 +174,13 @@ def populate_from_dict(self, data, keys=None, skip=None, track_changes=True): """ cls = type(self) changed = {} - for key, value in data.iteritems(): + for key, value in data.items(): if keys and key not in keys: continue if skip and key in skip: continue if not hasattr(cls, key): - raise ValueError("{} has no attribute '{}'".format(cls.__name__, key)) + raise ValueError(f"{cls.__name__} has no attribute '{key}'") if not track_changes: setattr(self, key, value) continue @@ -227,7 +194,7 @@ def populate_from_dict(self, data, keys=None, skip=None, track_changes=True): return changed if track_changes else None def populate_from_attrs(self, obj, attrs): - """Populates the object from another object's attributes + """Populate the object from another object's attributes. :param obj: an object :param attrs: a set containing the attributes to copy @@ -235,20 +202,19 @@ def populate_from_attrs(self, obj, attrs): cls = type(self) for attr in attrs: if not hasattr(cls, attr): - raise ValueError("{} has no attribute '{}'".format(cls.__name__, attr)) + raise ValueError(f"{cls.__name__} has no attribute '{attr}'") setattr(self, attr, getattr(obj, attr)) @listens_for(orm.mapper, 'after_configured', once=True) def _mappers_configured(): - from indico.core.db import db - for model in db.Model._decl_class_registry.itervalues(): + for model in get_all_models(): if hasattr(model, '__table__') and model.allow_relationship_preloading: listen(model, 'load', model._populate_preloaded_relationships) def import_all_models(package_name='indico'): - """Utility that imports all modules in indico/**/models/ + """Utility that imports all modules in indico/**/models/. :param package_name: Package name to scan for models. If unset, the top-level package containing this file @@ -261,7 +227,7 @@ def import_all_models(package_name='indico'): for root, dirs, files in os.walk(package_root): if os.path.basename(root) == 'models': package = os.path.relpath(root, package_root).replace(os.sep, '.') - modules += ['{}.{}.{}'.format(package_name, package, name[:-3]) + modules += [f'{package_name}.{package}.{name[:-3]}' for name in files if name.endswith('.py') and name != '__init__.py' and not name.endswith('_test.py')] @@ -269,8 +235,14 @@ def import_all_models(package_name='indico'): import_module(module) +def get_all_models(): + """Get all models SQLAlchemy knows about.""" + from indico.core.db import db + return {mapper.class_ for mapper in db.Model.registry.mappers} + + def attrs_changed(obj, *attrs): - """Checks if the given fields have been changed since the last flush + """Check if the given fields have been changed since the last flush. :param obj: SQLAlchemy-mapped object :param attrs: attribute names @@ -279,8 +251,10 @@ def attrs_changed(obj, *attrs): def get_default_values(model): - """Returns a dict containing all static default values of a model. + """Return a dict containing all static default values of a model. + This only takes `default` into account, not `server_default`. + :param model: A SQLAlchemy model """ return {attr.key: attr.columns[0].default.arg @@ -290,7 +264,7 @@ def get_default_values(model): def get_simple_column_attrs(model): """ - Returns a set containing all "simple" column attributes, i.e. + Return a set containing all "simple" column attributes, i.e. attributes which map to a table column and are neither primary key nor foreign key. @@ -308,7 +282,7 @@ def get_simple_column_attrs(model): def auto_table_args(cls, **extra_kwargs): - """Merges SQLAlchemy ``__table_args__`` values. + """Merge SQLAlchemy ``__table_args__`` values. This is useful when using mixins to compose model classes if the mixins need to define custom ``__table_args__``. Since defining @@ -346,7 +320,7 @@ def __table_args__(cls): else: posargs.extend(value) else: # pragma: no cover - raise ValueError('Unexpected tableargs: {}'.format(value)) + raise ValueError(f'Unexpected tableargs: {value}') kwargs.update(extra_kwargs) if posargs and kwargs: return tuple(posargs) + (kwargs,) @@ -357,11 +331,11 @@ def __table_args__(cls): def _get_backref_name(relationship): - return relationship.backref if isinstance(relationship.backref, basestring) else relationship.backref[0] + return relationship.backref if isinstance(relationship.backref, str) else relationship.backref[0] def populate_one_to_one_backrefs(model, *relationships): - """Populates the backref of a one-to-one relationship on load + """Populate the backref of a one-to-one relationship on load. See this post in the SQLAlchemy docs on why it's useful/necessary: http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#creating-custom-load-rules @@ -377,7 +351,7 @@ def _mappers_configured(): @listens_for(model, 'load') def _populate_backrefs(target, context): - for name, backref in mappings.iteritems(): + for name, backref in mappings.items(): # __dict__ to avoid triggering lazy-loaded relationships if target.__dict__.get(name) is not None: set_committed_value(getattr(target, name), backref, target) diff --git a/indico/core/db/sqlalchemy/util/models_test.py b/indico/core/db/sqlalchemy/util/models_test.py index b25a99e9e9a..1abbec7d203 100644 --- a/indico/core/db/sqlalchemy/util/models_test.py +++ b/indico/core/db/sqlalchemy/util/models_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -34,7 +34,7 @@ def test_auto_table_args(args, kw, expected): classes = [] for i, arg in enumerate(args): - name = 'Test{}'.format(i) - classes.append(type(name, (object,), {'_{}__auto_table_args'.format(name): arg})) + name = f'Test{i}' + classes.append(type(name, (object,), {f'_{name}__auto_table_args': arg})) cls = type('Test', tuple(classes), {}) assert auto_table_args(cls, **kw) == expected diff --git a/indico/core/db/sqlalchemy/util/queries.py b/indico/core/db/sqlalchemy/util/queries.py index 5db11b7ea15..25f91179589 100644 --- a/indico/core/db/sqlalchemy/util/queries.py +++ b/indico/core/db/sqlalchemy/util/queries.py @@ -1,24 +1,21 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import re from sqlalchemy import func, inspect, over from sqlalchemy.sql import update -TS_REGEX = re.compile(r'([@<>!()&|:\'])') +TS_REGEX = re.compile(r'([@<>!()&|:\'\\])') def limit_groups(query, model, partition_by, order_by, limit=None, offset=0): - """Limits the number of rows returned for each group - + """Limit the number of rows returned for each group. This utility allows you to apply a limit/offset to grouped rows of a query. Note that the query will only contain the data from `model`; i.e. you cannot @@ -51,7 +48,7 @@ def db_dates_overlap(entity, start_column, start, end_column, end, inclusive=Fal def escape_like(value): - """Escapes a string to be used as a plain string in LIKE""" + """Escape a string to be used as a plain string in LIKE.""" escape_char = '\\' return (value .replace(escape_char, escape_char * 2) # literal escape char needs to be escaped @@ -61,22 +58,29 @@ def escape_like(value): def preprocess_ts_string(text, prefix=True): atoms = [TS_REGEX.sub(r'\\\1', atom.strip()) for atom in text.split()] - return ' & '.join('{}:*'.format(atom) if prefix else atom for atom in atoms) + return ' & '.join(f'{atom}:*' if prefix else atom for atom in atoms) def has_extension(conn, name): - """Checks if the postgres database has a certain extension installed""" + """Check if the postgres database has a certain extension installed.""" return conn.execute("SELECT EXISTS(SELECT TRUE FROM pg_extension WHERE extname = %s)", (name,)).scalar() def get_postgres_version(): from indico.core.db import db version = db.engine.execute("SELECT current_setting('server_version_num')::int").scalar() - return '{}.{}.{}'.format(version // 10000, version % 10000 // 100, version % 100) + major = version // 10000 + minor = version % 10000 // 100 + patch = version % 100 + if major >= 10: + # https://www.postgresql-archive.org/PG-VERSION-NUM-formatted-incorrectly-td6002110.html + return f'{major}.{patch}' + else: + return f'{major}.{minor}.{patch}' def increment_and_get(col, filter_, n=1): - """Increments and returns a numeric column. + """Increment and returns a numeric column. This is committed to the database immediately in a separate transaction to avoid possible conflicts. @@ -117,16 +121,16 @@ def get_related_object(obj, relationship, criteria): such object could be found. """ def _compare(a, b): - if isinstance(a, basestring) and a.isdigit(): + if isinstance(a, str) and a.isdigit(): a = int(a) - if isinstance(b, basestring) and b.isdigit(): + if isinstance(b, str) and b.isdigit(): b = int(b) return a == b # if the relationship is loaded evaluate the criteria in python if relationship not in inspect(obj).unloaded: return next((x for x in getattr(obj, relationship) - if all(_compare(getattr(x, k), v) for k, v in criteria.iteritems())), + if all(_compare(getattr(x, k), v) for k, v in criteria.items())), None) # otherwise query that specific object cls = getattr(type(obj), relationship).prop.mapper.class_ @@ -152,7 +156,7 @@ def _get(): _offset[0] += limit return rv - results = filter(predicate, _get()) + results = list(filter(predicate, _get())) while len(results) < n: objects = _get() if not objects: diff --git a/indico/core/db/sqlalchemy/util/session.py b/indico/core/db/sqlalchemy/util/session.py index e561875eba4..3369b9776f9 100644 --- a/indico/core/db/sqlalchemy/util/session.py +++ b/indico/core/db/sqlalchemy/util/session.py @@ -1,19 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import wraps from indico.core.db import db def no_autoflush(fn): - """Wraps the decorated function in a no-autoflush block""" + """Wrap the decorated function in a no-autoflush block.""" @wraps(fn) def wrapper(*args, **kwargs): with db.session.no_autoflush: diff --git a/indico/core/emails.py b/indico/core/emails.py index 9962174a9db..aa2abd78244 100644 --- a/indico/core/emails.py +++ b/indico/core/emails.py @@ -1,29 +1,29 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import, unicode_literals - -import cPickle import os +import pickle import tempfile from datetime import date +from email.utils import make_msgid import click from celery.exceptions import MaxRetriesExceededError, Retry from sqlalchemy.orm.attributes import flag_modified +from werkzeug.urls import url_parse from indico.core.celery import celery from indico.core.config import config from indico.core.db import db from indico.core.logger import Logger from indico.util.date_time import now_utc -from indico.util.emails.backend import EmailBackend -from indico.util.emails.message import EmailMessage from indico.util.string import truncate +from indico.vendor.django_mail import get_connection +from indico.vendor.django_mail.message import EmailMessage logger = Logger.get('emails') @@ -81,7 +81,7 @@ def do_send_email(email, log_entry=None, _from_task=False): :param _from_task: Indicates that this function is called from the celery task responsible for sending emails. """ - with EmailBackend(timeout=config.SMTP_TIMEOUT) as conn: + with get_connection() as conn: msg = EmailMessage(subject=email['subject'], body=email['body'], from_email=email['from'], to=email['to'], cc=email['cc'], bcc=email['bcc'], reply_to=email['reply_to'], attachments=email['attachments'], connection=conn) @@ -89,6 +89,7 @@ def do_send_email(email, log_entry=None, _from_task=False): msg.extra_headers['To'] = 'Undisclosed-recipients:;' if email['html']: msg.content_subtype = 'html' + msg.extra_headers['message-id'] = make_msgid(domain=url_parse(config.BASE_URL).host) msg.send() if not _from_task: logger.info('Sent email "%s"', truncate(email['subject'], 100)) @@ -106,10 +107,10 @@ def update_email_log_state(log_entry, failed=False): def store_failed_email(email, log_entry=None): - prefix = 'failed-email-{}-'.format(date.today().isoformat()) + prefix = f'failed-email-{date.today().isoformat()}-' fd, name = tempfile.mkstemp(prefix=prefix, dir=config.TEMP_DIR) with os.fdopen(fd, 'wb') as f: - cPickle.dump((email, log_entry.id if log_entry else None), f) + pickle.dump((email, log_entry.id if log_entry else None), f) return name @@ -117,7 +118,7 @@ def resend_failed_email(path): """Try re-sending an email that previously failed.""" from indico.modules.events.logs import EventLogEntry with open(path, 'rb') as f: - email, log_entry_id = cPickle.load(f) + email, log_entry_id = pickle.load(f) log_entry = EventLogEntry.get(log_entry_id) if log_entry_id is not None else None do_send_email(email, log_entry) db.session.commit() diff --git a/indico/core/errors.py b/indico/core/errors.py index 06215d0efa6..10fccc5824d 100644 --- a/indico/core/errors.py +++ b/indico/core/errors.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -12,11 +12,10 @@ from werkzeug.exceptions import BadRequest, Forbidden, HTTPException, NotFound from indico.util.i18n import _ -from indico.util.string import to_unicode def get_error_description(exception): - """Gets a user-friendy description for an exception + """Get a user-friendy description for an exception. This overrides some HTTPException messages to be more suitable for end-users. @@ -24,15 +23,15 @@ def get_error_description(exception): try: description = exception.description except AttributeError: - return to_unicode(exception.message) + return str(exception) if isinstance(exception, Forbidden) and description == Forbidden.description: - return _(u"You are not allowed to access this page.") + return _("You are not allowed to access this page.") elif isinstance(exception, NotFound) and description == NotFound.description: - return _(u"The page you are looking for doesn't exist.") + return _("The page you are looking for doesn't exist.") elif isinstance(exception, BadRequest) and description == BadRequest.description: - return _(u"The request was invalid or contained invalid arguments.") + return _("The request was invalid or contained invalid arguments.") else: - return to_unicode(description) + return str(description) class IndicoError(Exception): diff --git a/indico/core/limiter.py b/indico/core/limiter.py new file mode 100644 index 00000000000..fd9381276d0 --- /dev/null +++ b/indico/core/limiter.py @@ -0,0 +1,78 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +import time +from datetime import timedelta + +from flask import has_request_context, request +from flask_limiter import Limiter +from limits import parse_many + + +class RateLimit: + def __init__(self, limiter, key_func, scope, limits): + self.limiter = limiter + self.key_func = key_func + self.scope = scope + self.limits = limits + + def _get_args(self): + return self.key_func(), self.scope + + def hit(self): + """Check the rate limit and increment the counter. + + This method should be used when performing an action that should + count against the user's rate limit. + """ + if self.limits is None: + return True + args = self._get_args() + return any(self.limiter.limiter.hit(lim, *args) for lim in self.limits) + + def test(self): + """Check the rate limit without incrementing the counter. + + This method should be used when you just want to see if the rate limit + has been triggered (e.g. to show a message when loading a form), without + counting the action against the rate limit. + """ + if self.limits is None: + return True + args = self._get_args() + return any(self.limiter.limiter.test(lim, *args) for lim in self.limits) + + def get_reset_delay(self): + """Get the duration until the rate limit resets.""" + if self.limits is None: + return timedelta() + args = self._get_args() + reset = min(self.limiter.limiter.get_window_stats(lim, *args)[0] for lim in self.limits) + return timedelta(seconds=reset - int(time.time())) + + def __repr__(self): + limits = '; '.join(str(lim) for lim in self.limits) if self.limits is not None else 'unlimited' + return f'' + + +def make_rate_limiter(scope, limits): + """Create a rate limiter. + + Multiple limits can be separated with a semicolon; in that case + all limits are checked until one succeeds. This allows specifying + a somewhat strict limit, but then a higher limit over a longer period + of time to allow for bursts. + """ + limits = list(parse_many(limits)) if limits is not None else None + return RateLimit(limiter, limiter._key_func, scope, limits) + + +def _limiter_key(): + return request.remote_addr if has_request_context() else 'dummy.ip' + + +limiter = Limiter(key_func=_limiter_key, strategy='moving-window', auto_check=False) diff --git a/indico/core/logger.py b/indico/core/logger.py index ce34d855eaa..c24da3a133e 100644 --- a/indico/core/logger.py +++ b/indico/core/logger.py @@ -1,58 +1,41 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import logging import logging.config import logging.handlers import os -import smtplib import warnings -from email.mime.text import MIMEText -from email.utils import formatdate from pprint import pformat import yaml -from flask import current_app, has_request_context, request, session +from flask import has_request_context, request, session from indico.core.config import config -from indico.util.i18n import set_best_lang -from indico.web.util import get_request_info - +from indico.web.util import get_request_info, get_request_user -try: - from raven import setup_logging - from raven.contrib.celery import register_logger_signal, register_signal - from raven.contrib.flask import Sentry - from raven.handlers.logging import SentryHandler -except ImportError: - Sentry = object # so we can subclass - has_sentry = False -else: - has_sentry = True - -class AddRequestIDFilter(object): +class AddRequestIDFilter: def filter(self, record): # Add our request ID if available record.request_id = request.id if has_request_context() else '0' * 16 return True -class AddUserIDFilter(object): +class AddUserIDFilter: def filter(self, record): - record.user_id = unicode(session.user.id) if has_request_context() and session and session.user else '-' + user = get_request_user()[0] if has_request_context() else None + record.user_id = str(session.user.id) if user else '-' return True class RequestInfoFormatter(logging.Formatter): def format(self, record): - rv = super(RequestInfoFormatter, self).format(record) + rv = super().format(record) info = get_request_info() if info: rv += '\n\n' + pformat(info) @@ -63,33 +46,6 @@ class FormattedSubjectSMTPHandler(logging.handlers.SMTPHandler): def getSubject(self, record): return self.subject % record.__dict__ - def emit(self, record): - # This is the same as the original SMTPHandler's method, but - # using MIMEText instead of string operations (which are not - # compatible with unicode) - try: - port = self.mailport - if not port: - port = smtplib.SMTP_PORT - smtp = smtplib.SMTP(self.mailhost, port, timeout=self._timeout) - msg = MIMEText(self.format(record), 'plain', 'utf-8') - msg['From'] = self.fromaddr - msg['To'] = ', '.join(self.toaddrs) - msg['Subject'] = self.getSubject(record) - msg['Date'] = formatdate() - if self.username: - if self.secure is not None: - smtp.ehlo() - smtp.starttls(*self.secure) - smtp.ehlo() - smtp.login(self.username, self.password) - smtp.sendmail(self.fromaddr, self.toaddrs, msg.as_string()) - smtp.quit() - except (KeyboardInterrupt, SystemExit): - raise - except Exception: - self.handleError(record) - class BlacklistFilter(logging.Filter): def __init__(self, names): @@ -99,7 +55,7 @@ def filter(self, record): return not any(x.filter(record) for x in self.filters) -class Logger(object): +class Logger: @classmethod def init(cls, app): path = config.LOGGING_CONFIG_PATH @@ -119,7 +75,7 @@ def init(cls, app): data.setdefault('filters', {}) data['filters']['_add_request_id'] = {'()': AddRequestIDFilter} data['filters']['_add_user_id'] = {'()': AddUserIDFilter} - for handler in data['handlers'].itervalues(): + for handler in data['handlers'].values(): handler.setdefault('filters', []) handler['filters'].insert(0, '_add_request_id') handler['filters'].insert(1, '_add_user_id') @@ -133,13 +89,13 @@ def init(cls, app): handler['secure'] = () if config.SMTP_USE_TLS else None # yuck, empty tuple == STARTTLS if config.SMTP_LOGIN and config.SMTP_PASSWORD: handler['credentials'] = (config.SMTP_LOGIN, config.SMTP_PASSWORD) - handler.setdefault('fromaddr', 'logger@{}'.format(config.WORKER_NAME)) + handler.setdefault('fromaddr', f'logger@{config.WORKER_NAME}') handler.setdefault('toaddrs', [config.SUPPORT_EMAIL]) subject = ('Unexpected Exception occurred at {}: %(message)s' if handler['class'] == 'indico.core.logger.FormattedSubjectSMTPHandler' else 'Unexpected Exception occurred at {}') handler.setdefault('subject', subject.format(config.WORKER_NAME)) - for formatter in data['formatters'].itervalues(): + for formatter in data['formatters'].values(): # Make adding request info to log entries less ugly if formatter.pop('append_request_info', False): assert '()' not in formatter @@ -152,10 +108,6 @@ def init(cls, app): if config.CUSTOMIZATION_DEBUG and config.CUSTOMIZATION_DIR: data['loggers'].setdefault('indico.customization', {})['level'] = 'DEBUG' logging.config.dictConfig(data) - if config.SENTRY_DSN: - if not has_sentry: - raise Exception('`raven` must be installed to use sentry logging') - init_sentry(app) @classmethod def get(cls, name=None): @@ -170,71 +122,3 @@ def get(cls, name=None): elif name != 'indico' and not name.startswith('indico.'): name = 'indico.' + name return logging.getLogger(name) - - -class IndicoSentry(Sentry): - def get_user_info(self, request): - if not has_request_context() or not session.user: - return None - return {'id': session.user.id, - 'email': session.user.email, - 'name': session.user.full_name} - - def before_request(self, *args, **kwargs): - super(IndicoSentry, self).before_request() - if not has_request_context(): - return - self.client.extra_context({'Endpoint': str(request.url_rule.endpoint) if request.url_rule else None, - 'Request ID': request.id}) - self.client.tags_context({'locale': set_best_lang()}) - - -def init_sentry(app): - sentry = IndicoSentry(wrap_wsgi=False, register_signal=True, logging=False) - sentry.init_app(app) - # setup logging manually and exclude uncaught indico exceptions. - # these are logged manually in the flask error handler logic so - # we get the X-Sentry-ID header which is not populated in the - # logging handlers - handler = SentryHandler(sentry.client, level=getattr(logging, config.SENTRY_LOGGING_LEVEL)) - handler.addFilter(BlacklistFilter({'indico.flask', 'celery.redirected'})) - setup_logging(handler) - # connect to the celery logger - register_logger_signal(sentry.client) - register_signal(sentry.client) - - -def sentry_log_exception(): - try: - sentry = current_app.extensions['sentry'] - except KeyError: - return - sentry.captureException() - - -def sentry_set_extra(data): - """ - Set extra data to be logged in sentry if the current request - results in something to be sent to sentry. - - :param data: A dict containing data. - """ - try: - sentry = current_app.extensions['sentry'] - except KeyError: - return - sentry.client.extra_context(data) - - -def sentry_set_tags(data): - """ - Set extra tag data to be logged in sentry if the current request - results in something to be sent to sentry. - - :param data: A dict containing tag data. - """ - try: - sentry = current_app.extensions['sentry'] - except KeyError: - return - sentry.client.tags_context(data) diff --git a/indico/core/marshmallow.py b/indico/core/marshmallow.py index c62b8a128e7..c34f1f27b55 100644 --- a/indico/core/marshmallow.py +++ b/indico/core/marshmallow.py @@ -1,20 +1,18 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import, unicode_literals - from inspect import getmro from flask_marshmallow import Marshmallow -from flask_marshmallow.sqla import SchemaOpts +from flask_marshmallow.sqla import SQLAlchemyAutoSchemaOpts from marshmallow import fields, post_dump, post_load, pre_load from marshmallow_enum import EnumField from marshmallow_sqlalchemy import ModelConverter -from marshmallow_sqlalchemy import ModelSchema as MSQLAModelSchema +from marshmallow_sqlalchemy import SQLAlchemyAutoSchema as MSQLASQLAlchemyAutoSchema from sqlalchemy.orm import ColumnProperty from sqlalchemy.sql.elements import Label from webargs.flaskparser import parser as webargs_flask_parser @@ -39,7 +37,7 @@ class IndicoModelConverter(ModelConverter): }) def _get_field_kwargs_for_property(self, prop): - kwargs = super(IndicoModelConverter, self)._get_field_kwargs_for_property(prop) + kwargs = super()._get_field_kwargs_for_property(prop) if isinstance(prop, ColumnProperty) and hasattr(prop.columns[0].type, 'marshmallow_get_field_kwargs'): kwargs.update(prop.columns[0].type.marshmallow_get_field_kwargs()) return kwargs @@ -48,7 +46,7 @@ def _should_exclude_field(self, column, fields=None, exclude=None): if _is_column_property(column): # column_property isn't support and fails later, so we always exclude those return True - return super(IndicoModelConverter, self)._should_exclude_field(column, fields=fields, exclude=exclude) + return super()._should_exclude_field(column, fields=fields, exclude=exclude) def fields_for_model(self, model, *args, **kwargs): # Look up aliases on all classes in the inheritance chain of @@ -65,7 +63,7 @@ def _get_from_mro(attr, key, default=None, _mro=getmro(model)): # generate all fields from the models and leave it up to mm itself to # exclude fields we don't care about kwargs['fields'] = () - fields = super(IndicoModelConverter, self).fields_for_model(model, *args, **kwargs) + fields = super().fields_for_model(model, *args, **kwargs) # remove column_property leftovers so they don't break things when using # the schema without restricting the list of fields (it would still include @@ -74,7 +72,7 @@ def _get_from_mro(attr, key, default=None, _mro=getmro(model)): if _is_column_property(prop): del fields[prop.key] - for key, field in fields.items(): + for key, field in list(fields.items()): new_key = _get_from_mro('marshmallow_aliases', key) if new_key: del fields[key] @@ -86,7 +84,7 @@ def _get_from_mro(attr, key, default=None, _mro=getmro(model)): class IndicoSchema(mm.Schema): @post_dump(pass_many=True, pass_original=True) - def _call_post_dump_signal(self, data, many, orig, **kwargs): + def _call_post_dump_signal(self, data, orig, *, many, **kwargs): data_list = data if many else [data] orig_list = orig if many else [orig] signals.plugin.schema_post_dump.send(type(self), data=data_list, orig=orig_list, many=many) @@ -103,17 +101,18 @@ def _call_post_load_signal(self, data, **kwargs): return data -class _IndicoModelSchemaOpts(SchemaOpts): +class _IndicoSQLAlchemyAutoSchemaOpts(SQLAlchemyAutoSchemaOpts): def __init__(self, meta, **kwargs): - super(_IndicoModelSchemaOpts, self).__init__(meta, **kwargs) + super().__init__(meta, **kwargs) self.model_converter = getattr(meta, 'model_converter', IndicoModelConverter) + self.include_relationships = getattr(meta, 'include_relationships', True) -class IndicoModelSchema(MSQLAModelSchema, IndicoSchema): - OPTIONS_CLASS = _IndicoModelSchemaOpts +class IndicoSQLAlchemyAutoSchema(MSQLASQLAlchemyAutoSchema, IndicoSchema): + OPTIONS_CLASS = _IndicoSQLAlchemyAutoSchemaOpts mm.Schema = IndicoSchema -mm.ModelSchema = IndicoModelSchema +mm.SQLAlchemyAutoSchema = IndicoSQLAlchemyAutoSchema webargs_flask_parser.schema_class = IndicoSchema # just in case someone uses the wrong import indico_webargs_flask_parser.schema_class = IndicoSchema diff --git a/indico/core/notifications.py b/indico/core/notifications.py index 4b76bf9c8a4..64da12831fc 100644 --- a/indico/core/notifications.py +++ b/indico/core/notifications.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import re import time from functools import wraps @@ -17,7 +15,7 @@ from indico.core.config import config from indico.core.db import db from indico.core.logger import Logger -from indico.util.string import to_unicode, truncate +from indico.util.string import truncate logger = Logger.get('emails') @@ -33,13 +31,13 @@ def wrapper(*args, **kwargs): mails = list(mails) elif not isinstance(mails, list): mails = [mails] - for mail in filter(None, mails): + for mail in [_f for _f in mails if _f]: send_email(mail) return wrapper def send_email(email, event=None, module=None, user=None, log_metadata=None): - """Sends an email created by :func:`make_email`. + """Send an email created by :func:`make_email`. When called while inside a RH, the email will be queued and only sent or passed on to celery once the database commit succeeded. @@ -77,7 +75,7 @@ def _log_email(email, event, module, user, meta=None): 'state': 'pending', 'sent_dt': None, } - return event.log(EventLogRealm.emails, EventLogKind.other, to_unicode(module or 'Unknown'), log_data['subject'], + return event.log(EventLogRealm.emails, EventLogKind.other, module or 'Unknown', log_data['subject'], user, type_='email', data=log_data, meta=meta) @@ -122,7 +120,7 @@ def flush_email_queue(): def make_email(to_list=None, cc_list=None, bcc_list=None, from_address=None, reply_address=None, attachments=None, subject=None, body=None, template=None, html=False): - """Creates an email. + """Create an email. The preferred way to specify the email content is using the `template` argument. To do so, use :func:`.get_template_module` on @@ -161,18 +159,18 @@ def make_email(to_list=None, cc_list=None, bcc_list=None, from_address=None, rep cc_list = set() if bcc_list is None: bcc_list = set() - to_list = {to_list} if isinstance(to_list, basestring) else to_list - cc_list = {cc_list} if isinstance(cc_list, basestring) else cc_list - bcc_list = {bcc_list} if isinstance(bcc_list, basestring) else bcc_list - reply_address = {reply_address} if isinstance(reply_address, basestring) else (reply_address or set()) + to_list = {to_list} if isinstance(to_list, str) else to_list + cc_list = {cc_list} if isinstance(cc_list, str) else cc_list + bcc_list = {bcc_list} if isinstance(bcc_list, str) else bcc_list + reply_address = {reply_address} if isinstance(reply_address, str) else (reply_address or set()) return { - 'to': set(map(to_unicode, to_list)), - 'cc': set(map(to_unicode, cc_list)), - 'bcc': set(map(to_unicode, bcc_list)), - 'from': to_unicode(from_address or config.NO_REPLY_EMAIL), - 'reply_to': set(map(to_unicode, reply_address)), + 'to': set(to_list), + 'cc': set(cc_list), + 'bcc': set(bcc_list), + 'from': from_address or config.NO_REPLY_EMAIL, + 'reply_to': set(reply_address), 'attachments': attachments or [], - 'subject': to_unicode(subject).strip(), - 'body': to_unicode(body).strip(), + 'subject': subject.strip(), + 'body': body.strip(), 'html': html, } diff --git a/indico/core/oauth/__init__.py b/indico/core/oauth/__init__.py new file mode 100644 index 00000000000..9bfff18a70c --- /dev/null +++ b/indico/core/oauth/__init__.py @@ -0,0 +1,39 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +import os + +from indico.core import signals +from indico.core.db import db + +from .logger import logger +from .oauth2 import require_oauth + + +__all__ = ['require_oauth'] + + +@signals.app_created.connect +def _no_ssl_required_on_debug(app, **kwargs): + if app.debug or app.testing: + os.environ['AUTHLIB_INSECURE_TRANSPORT'] = '1' + + +@signals.users.merged.connect +def _delete_merged_user_tokens(target, source, **kwargs): + target_app_links = {link.application: link for link in target.oauth_app_links} + for source_link in source.oauth_app_links.all(): + try: + target_link = target_app_links[source_link.application] + except KeyError: + logger.info('merge: reassigning %r to %r', source_link, target) + source_link.user = target + else: + logger.info('merge: merging %r into %r', source_link, target_link) + target_link.update_scopes(set(source_link.scopes)) + target_link.tokens.extend(source_link.tokens) + db.session.delete(source_link) diff --git a/indico/core/oauth/endpoints.py b/indico/core/oauth/endpoints.py new file mode 100644 index 00000000000..eb2ef958216 --- /dev/null +++ b/indico/core/oauth/endpoints.py @@ -0,0 +1,48 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from authlib.oauth2.rfc7009 import RevocationEndpoint +from authlib.oauth2.rfc7662 import IntrospectionEndpoint + +from indico.core.config import config +from indico.core.db import db +from indico.core.oauth.logger import logger +from indico.core.oauth.models.tokens import OAuthToken +from indico.core.oauth.util import query_token + + +class IndicoIntrospectionEndpoint(IntrospectionEndpoint): + SUPPORTED_TOKEN_TYPES = ('access_token',) + CLIENT_AUTH_METHODS = ('client_secret_basic', 'client_secret_post') + + def check_permission(self, token, client, request): + return token.check_client(client) + + def query_token(self, token_string, token_type_hint): + return query_token(token_string) + + def introspect_token(self, token: OAuthToken): + return { + 'active': True, + 'client_id': token.application.client_id, + 'token_type': 'Bearer', + 'scope': token.get_scope(), + 'sub': str(token.user.id), + 'iss': config.BASE_URL + } + + +class IndicoRevocationEndpoint(RevocationEndpoint): + SUPPORTED_TOKEN_TYPES = ('access_token',) + CLIENT_AUTH_METHODS = ('client_secret_basic', 'client_secret_post') + + def query_token(self, token, token_type_hint): + return query_token(token) + + def revoke_token(self, token, request): + db.session.delete(token) + logger.info('Token %s was revoked', token) diff --git a/indico/core/oauth/grants.py b/indico/core/oauth/grants.py new file mode 100644 index 00000000000..1b2dcae2933 --- /dev/null +++ b/indico/core/oauth/grants.py @@ -0,0 +1,62 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from authlib.oauth2.rfc6749 import InvalidScopeError, scope_to_list +from authlib.oauth2.rfc6749.grants import AuthorizationCodeGrant +from authlib.oauth2.rfc7636.challenge import CodeChallenge + +from indico.core.cache import make_scoped_cache +from indico.core.oauth.models.tokens import OAuth2AuthorizationCode +from indico.modules.users import User + + +auth_code_store = make_scoped_cache('oauth-grant-tokens') + + +class IndicoAuthorizationCodeGrant(AuthorizationCodeGrant): + TOKEN_ENDPOINT_AUTH_METHODS = ('client_secret_basic', 'client_secret_post', 'none') + + def save_authorization_code(self, code, request): + code_challenge = request.data.get('code_challenge') + code_challenge_method = request.data.get('code_challenge_method') + auth_code = OAuth2AuthorizationCode( + code=code, + client_id=request.client.client_id, + redirect_uri=request.redirect_uri, + scope=request.scope, + user_id=request.user.id, + code_challenge=code_challenge, + code_challenge_method=code_challenge_method, + ) + auth_code_store.add(code, auth_code, 3600) + return auth_code + + def query_authorization_code(self, code, client): + auth_code = auth_code_store.get(code) + if auth_code and auth_code.client_id == client.client_id and not auth_code.is_expired(): + return auth_code + + def delete_authorization_code(self, authorization_code): + auth_code_store.delete(authorization_code.code) + + def authenticate_user(self, authorization_code): + return User.get(authorization_code.user_id, is_deleted=False) + + def validate_requested_scope(self): + """Validate if requested scope is supported by Authorization Server.""" + scope = self.request.scope + state = self.request.state + if scope: + allowed = set(scope_to_list(self.request.client.get_allowed_scope(scope))) + requested = set(scope_to_list(scope)) + if not (requested <= allowed): + raise InvalidScopeError(state=state) + return self.server.validate_requested_scope(scope, state) + + +class IndicoCodeChallenge(CodeChallenge): + SUPPORTED_CODE_CHALLENGE_METHOD = ('S256',) diff --git a/indico/legacy/services/interface/rpc/handlers.py b/indico/core/oauth/logger.py similarity index 50% rename from indico/legacy/services/interface/rpc/handlers.py rename to indico/core/oauth/logger.py index 165a5fc49dc..07234407bcf 100644 --- a/indico/legacy/services/interface/rpc/handlers.py +++ b/indico/core/oauth/logger.py @@ -1,14 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from importlib import import_module +from indico.core.logger import Logger -methodMap = {} -endpointMap = { - "search": import_module('indico.legacy.services.implementation.search') -} +logger = Logger.get('oauth') diff --git a/indico/legacy/fossils/__init__.py b/indico/core/oauth/models/__init__.py similarity index 100% rename from indico/legacy/fossils/__init__.py rename to indico/core/oauth/models/__init__.py diff --git a/indico/core/oauth/models/applications.py b/indico/core/oauth/models/applications.py new file mode 100644 index 00000000000..25359b720c9 --- /dev/null +++ b/indico/core/oauth/models/applications.py @@ -0,0 +1,258 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from uuid import uuid4 + +from authlib.oauth2.rfc6749 import ClientMixin, list_to_scope, scope_to_list +from sqlalchemy.dialects.postgresql import ARRAY, UUID +from sqlalchemy.ext.declarative import declared_attr +from werkzeug.urls import url_parse + +from indico.core.db import db +from indico.core.db.sqlalchemy import PyIntEnum +from indico.core.oauth.logger import logger +from indico.util.enum import IndicoEnum + + +class SystemAppType(int, IndicoEnum): + none = 0 + checkin = 1 + + __enforced_data__ = { + checkin: {'allowed_scopes': {'registrants'}, + 'redirect_uris': ['http://localhost'], + 'allow_pkce_flow': True, + 'is_enabled': True}, + } + + __default_data__ = { + checkin: {'is_trusted': True, + 'name': 'Checkin App', + 'description': 'The checkin app for mobile devices allows scanning ticket QR codes and ' + 'checking-in event participants.'}, + } + + @property + def enforced_data(self): + return self.__enforced_data__.get(self, {}) + + @property + def default_data(self): + return dict(self.__default_data__.get(self, {}), **self.enforced_data) + + +class OAuthApplication(ClientMixin, db.Model): + """OAuth applications registered in Indico.""" + + __tablename__ = 'applications' + + @declared_attr + def __table_args__(cls): + return (db.Index('ix_uq_applications_name_lower', db.func.lower(cls.name), unique=True), + db.Index(None, cls.system_app_type, unique=True, + postgresql_where=db.text(f'system_app_type != {SystemAppType.none.value}')), + {'schema': 'oauth'}) + + #: the unique id of the application + id = db.Column( + db.Integer, + primary_key=True + ) + #: human readable name + name = db.Column( + db.String, + nullable=False + ) + #: human readable description + description = db.Column( + db.Text, + nullable=False, + default='' + ) + #: the OAuth client_id + client_id = db.Column( + UUID, + unique=True, + nullable=False, + default=lambda: str(uuid4()) + ) + #: the OAuth client_secret + client_secret = db.Column( + UUID, + nullable=False, + default=lambda: str(uuid4()) + ) + #: the OAuth scopes the application may request access to + allowed_scopes = db.Column( + ARRAY(db.String), + nullable=False + ) + #: the OAuth absolute URIs that a application may use to redirect to after authorization + redirect_uris = db.Column( + ARRAY(db.String), + nullable=False, + default=[] + ) + #: whether the application is enabled or disabled + is_enabled = db.Column( + db.Boolean, + nullable=False, + default=True + ) + #: whether the application can access user data without asking for permission + is_trusted = db.Column( + db.Boolean, + nullable=False, + default=False + ) + #: whether the application can use the PKCE flow without a client secret + allow_pkce_flow = db.Column( + db.Boolean, + nullable=False, + default=False + ) + #: the type of system app (if any). system apps cannot be deleted + system_app_type = db.Column( + PyIntEnum(SystemAppType), + nullable=False, + default=SystemAppType.none + ) + + # relationship backrefs: + # - user_links (OAuthApplicationUserLink.application) + + @property + def default_redirect_uri(self): + return self.redirect_uris[0] if self.redirect_uris else None + + @property + def locator(self): + return {'id': self.id} + + def __repr__(self): # pragma: no cover + return f'' + + def reset_client_secret(self): + self.client_secret = str(uuid4()) + logger.info("Client secret for %s has been reset.", self) + + def get_client_id(self): + return self.client_id + + def get_default_redirect_uri(self): + return self.default_redirect_uri + + def get_allowed_scope(self, scope): + if not scope: + return '' + allowed = set(self.allowed_scopes) + scopes = set(scope_to_list(scope)) + return list_to_scope(allowed & scopes) + + def check_redirect_uri(self, redirect_uri): + """Called by authlib to validate the redirect_uri. + + Uses a logic similar to the one at GitHub, i.e. protocol and + host/port must match exactly and if there is a path in the + whitelisted URL, the path of the redirect_uri must start with + that path. + """ + # TODO: maybe use a stricter implementation that does not use substrings? + uri_data = url_parse(redirect_uri) + for valid_uri_data in map(url_parse, self.redirect_uris): + if (uri_data.scheme == valid_uri_data.scheme and uri_data.netloc == valid_uri_data.netloc and + uri_data.path.startswith(valid_uri_data.path)): + return True + return False + + def check_client_secret(self, client_secret): + return self.client_secret == client_secret + + def check_endpoint_auth_method(self, method, endpoint): + from indico.core.oauth.endpoints import IndicoIntrospectionEndpoint, IndicoRevocationEndpoint + from indico.core.oauth.grants import IndicoAuthorizationCodeGrant + + if endpoint == 'token': + if method == 'none' and not self.allow_pkce_flow: + return False + return method in IndicoAuthorizationCodeGrant.TOKEN_ENDPOINT_AUTH_METHODS + elif endpoint == 'introspection': + return method in IndicoIntrospectionEndpoint.CLIENT_AUTH_METHODS + elif endpoint == 'revocation': + return method in IndicoRevocationEndpoint.CLIENT_AUTH_METHODS + + # authlib returns True for unhandled cases, but since we do not have any other endpoints + # I'd rather fail and implement other cases as needed instead of silently accepting + # everything + raise NotImplementedError + + def check_response_type(self, response_type): + # We no longer allow the implicit flow, so `code` is all we need + return response_type == 'code' + + def check_grant_type(self, grant_type): + return grant_type == 'authorization_code' + + +class OAuthApplicationUserLink(db.Model): + """The authorization link between an OAuth app and a user.""" + + __tablename__ = 'application_user_links' + __table_args__ = (db.UniqueConstraint('application_id', 'user_id'), + {'schema': 'oauth'}) + + id = db.Column( + db.Integer, + primary_key=True + ) + application_id = db.Column( + db.Integer, + db.ForeignKey('oauth.applications.id', ondelete='CASCADE'), + nullable=False, + index=True + ) + user_id = db.Column( + db.Integer, + db.ForeignKey('users.users.id', ondelete='CASCADE'), + nullable=False, + index=True + ) + scopes = db.Column( + ARRAY(db.String), + nullable=False, + default=[] + ) + + application = db.relationship( + 'OAuthApplication', + lazy=False, + backref=db.backref( + 'user_links', + lazy='dynamic', + cascade='all, delete-orphan', + passive_deletes=True + ) + ) + user = db.relationship( + 'User', + lazy=True, + backref=db.backref( + 'oauth_app_links', + lazy='dynamic', + cascade='all, delete-orphan', + passive_deletes=True + ) + ) + + # relationship backrefs: + # - tokens (OAuthToken.app_user_link) + + def __repr__(self): + return f'' + + def update_scopes(self, scopes: set): + self.scopes = sorted(set(self.scopes) | scopes) diff --git a/indico/modules/oauth/models/applications_test.py b/indico/core/oauth/models/applications_test.py similarity index 83% rename from indico/modules/oauth/models/applications_test.py rename to indico/core/oauth/models/applications_test.py index a7563e39456..cfdb1138a40 100644 --- a/indico/modules/oauth/models/applications_test.py +++ b/indico/core/oauth/models/applications_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -8,11 +8,7 @@ import pytest -pytest_plugins = 'indico.modules.oauth.testing.fixtures' - - -def test_client_type(dummy_application): - assert dummy_application.client_type == 'public' +pytest_plugins = 'indico.core.oauth.testing.fixtures' @pytest.mark.parametrize(('redirect_uris', 'expected'), ( @@ -47,6 +43,6 @@ def test_reset_client_secret(dummy_application): (['https://test.com/a/b'], 'https://test.com/a', False), (['https://test.com/a/b', 'https://test2.com'], 'https://test2.com', True), )) -def test_validate_redirect_uri(dummy_application, redirect_uris, to_validate, validates): +def test_check_redirect_uri(dummy_application, redirect_uris, to_validate, validates): dummy_application.redirect_uris = redirect_uris - assert dummy_application.validate_redirect_uri(to_validate) == validates + assert dummy_application.check_redirect_uri(to_validate) == validates diff --git a/indico/core/oauth/models/tokens.py b/indico/core/oauth/models/tokens.py new file mode 100644 index 00000000000..1de7adb3f3d --- /dev/null +++ b/indico/core/oauth/models/tokens.py @@ -0,0 +1,136 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from dataclasses import dataclass, field +from datetime import datetime, timedelta + +from authlib.oauth2.rfc6749 import list_to_scope +from authlib.oauth2.rfc6749.models import AuthorizationCodeMixin, TokenMixin +from sqlalchemy.dialects.postgresql import ARRAY + +from indico.core.db import db +from indico.core.db.sqlalchemy import UTCDateTime +from indico.util.date_time import now_utc +from indico.util.passwords import TokenProperty + + +class OAuthToken(TokenMixin, db.Model): + """OAuth tokens.""" + + __tablename__ = 'tokens' + __table_args__ = {'schema': 'oauth'} + + id = db.Column( + db.Integer, + primary_key=True + ) + app_user_link_id = db.Column( + db.ForeignKey('oauth.application_user_links.id', ondelete='CASCADE'), + nullable=False, + index=True + ) + access_token_hash = db.Column( + db.String, + unique=True, + index=True, + nullable=False + ) + _scopes = db.Column( + 'scopes', + ARRAY(db.String) + ) + created_dt = db.Column( + UTCDateTime, + nullable=False, + default=now_utc + ) + last_used_dt = db.Column( + UTCDateTime, + nullable=True + ) + + access_token = TokenProperty('access_token_hash') + + app_user_link = db.relationship( + 'OAuthApplicationUserLink', + lazy=False, + backref=db.backref( + 'tokens', + lazy='dynamic', + cascade='all, delete-orphan', + passive_deletes=True + ) + ) + + @property + def user(self): + return self.app_user_link.user + + @property + def application(self): + return self.app_user_link.application + + @property + def locator(self): + return {'id': self.id} + + @property + def scopes(self): + """The set of scopes the linked application has access to.""" + return set(self._scopes) + + @scopes.setter + def scopes(self, value): + self._scopes = sorted(value) + + def __repr__(self): # pragma: no cover + return f'' + + def check_client(self, client): + return self.application == client + + def get_scope(self): + # scopes are restricted by what's authorized for the particular user and what's whitelisted for the app + scopes = self.scopes & set(self.app_user_link.scopes) & set(self.application.allowed_scopes) + return list_to_scope(sorted(scopes)) + + def get_expires_in(self): + return 0 + + def is_expired(self): + return False + + def is_revoked(self): + return self.user.is_blocked or self.user.is_deleted or not self.application.is_enabled + + +@dataclass(frozen=True) +class OAuth2AuthorizationCode(AuthorizationCodeMixin): + code: str + user_id: int + client_id: str + code_challenge: str + code_challenge_method: str + redirect_uri: str = '' + scope: str = '' + auth_time: datetime = field(default_factory=now_utc) + + def is_expired(self): + return now_utc() - self.auth_time > timedelta(minutes=5) + + def get_redirect_uri(self): + return self.redirect_uri + + def get_scope(self): + return self.scope + + def get_auth_time(self): + return self.auth_time + + def get_nonce(self): + # our grant types do not require nonces + raise NotImplementedError diff --git a/indico/core/oauth/models/tokens_test.py b/indico/core/oauth/models/tokens_test.py new file mode 100644 index 00000000000..4b0d8066bd3 --- /dev/null +++ b/indico/core/oauth/models/tokens_test.py @@ -0,0 +1,24 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +pytest_plugins = 'indico.core.oauth.testing.fixtures' + + +def test_token_locator(dummy_token): + assert dummy_token.locator == {'id': dummy_token.id} + + +def test_token_expires(dummy_token): + assert dummy_token.get_expires_in() == 0 + + +def test_token_scopes(dummy_token): + assert dummy_token.scopes == set(dummy_token._scopes) + new_scopes = ['c', 'b', 'a'] + dummy_token.scopes = new_scopes + assert dummy_token._scopes == sorted(new_scopes) + assert dummy_token.scopes == set(new_scopes) diff --git a/indico/core/oauth/oauth2.py b/indico/core/oauth/oauth2.py new file mode 100644 index 00000000000..a349abd2ce0 --- /dev/null +++ b/indico/core/oauth/oauth2.py @@ -0,0 +1,33 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from authlib.integrations.flask_oauth2 import AuthorizationServer + +from indico.core.oauth.endpoints import IndicoIntrospectionEndpoint, IndicoRevocationEndpoint +from indico.core.oauth.grants import IndicoAuthorizationCodeGrant, IndicoCodeChallenge +from indico.core.oauth.protector import IndicoAuthlibHTTPError, IndicoBearerTokenValidator, IndicoResourceProtector +from indico.core.oauth.scopes import SCOPES +from indico.core.oauth.util import query_client, save_token + + +auth_server = AuthorizationServer(query_client=query_client, save_token=save_token) +require_oauth = IndicoResourceProtector() + + +def setup_oauth_provider(app): + app.config.update({ + 'OAUTH2_SCOPES_SUPPORTED': list(SCOPES), + 'OAUTH2_TOKEN_EXPIRES_IN': { + 'authorization_code': 0, + } + }) + app.register_error_handler(IndicoAuthlibHTTPError, lambda exc: exc.get_response()) + auth_server.init_app(app) + auth_server.register_grant(IndicoAuthorizationCodeGrant, [IndicoCodeChallenge(required=True)]) + auth_server.register_endpoint(IndicoIntrospectionEndpoint) + auth_server.register_endpoint(IndicoRevocationEndpoint) + require_oauth.register_token_validator(IndicoBearerTokenValidator()) diff --git a/indico/core/oauth/oauth2_test.py b/indico/core/oauth/oauth2_test.py new file mode 100644 index 00000000000..19011582c25 --- /dev/null +++ b/indico/core/oauth/oauth2_test.py @@ -0,0 +1,457 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +import hashlib +from base64 import b64encode +from unittest.mock import MagicMock + +import pytest +from authlib.common.security import generate_token +from authlib.oauth2.client import OAuth2Client +from sqlalchemy.dialects.postgresql.array import ARRAY +from werkzeug.urls import url_parse + +from indico.core.oauth.models.applications import OAuthApplicationUserLink +from indico.core.oauth.models.tokens import OAuthToken +from indico.core.oauth.scopes import SCOPES +from indico.core.oauth.util import MAX_TOKENS_PER_SCOPE, save_token +from indico.modules.users.util import merge_users +from indico.web.flask.util import url_for + + +pytest_plugins = 'indico.core.oauth.testing.fixtures' + + +class MockSession: + def __init__(self, client): + self.client = client + + def request(self, method, url, *, data, headers, auth): + # we need to copy the headers since `prepare()` may modify them in-place, + # and the initial header dict may be coming from the client's DEFAULT_HEADERS, + # so without the copy we'd add basic auth headers to the default affecting future + # requests. usually requests does that, but since we pretend to be requests but + # use the flask test client instead, we need to take care of making the copy... + headers = headers.copy() + url, headers, data = auth.prepare(method, url, headers, data) + return CallableJsonWrapper(self.client.open(url, method=method, data=data, headers=headers)) + + +class CallableJsonWrapper: + def __init__(self, resp): + self.resp = resp + + def json(self): + return self.resp.json + + def __getattr__(self, attr): + return getattr(self.resp, attr) + + +@pytest.mark.parametrize('trusted', (False, True)) +@pytest.mark.parametrize(('token_endpoint_auth_method', 'pkce'), ( + ('client_secret_basic', False), + ('client_secret_post', False), + ('none', True), +)) +def test_oauth_flows(create_application, test_client, dummy_user, app, trusted, token_endpoint_auth_method, pkce): + oauth_app = create_application(name='test', is_trusted=trusted) + oauth_client = OAuth2Client(MockSession(test_client), + oauth_app.client_id, + oauth_app.client_secret if not pkce else None, + code_challenge_method=('S256' if pkce else None), + scope='read:user', + response_type='code', + token_endpoint=url_for('oauth.oauth_token', _external=True), + token_endpoint_auth_method=token_endpoint_auth_method, + redirect_uri=oauth_app.default_redirect_uri) + + code_verifier = generate_token(64) if pkce else None + auth_url, state = oauth_client.create_authorization_url(url_for('oauth.oauth_authorize', _external=True), + code_verifier=code_verifier) + + with test_client.session_transaction() as sess: + sess.set_session_user(dummy_user) + + # get consent page + resp = test_client.get(auth_url) + if trusted: + authorized_resp = resp + else: + assert resp.status_code == 200 + assert b'is requesting the following permissions' in resp.data + assert b'User information (read only)' in resp.data + + # give consent + authorized_resp = test_client.post(auth_url, data={'confirm': '1'}) + + assert authorized_resp.status_code == 302 + target_url = authorized_resp.headers['Location'] + target_url_parts = url_parse(target_url) + + assert f'state={state}' in target_url_parts.query + assert target_url == f'{oauth_app.default_redirect_uri}?{target_url_parts.query}' + + # get a token and make sure it looks fine + token = oauth_client.fetch_token(authorization_response=target_url, code_verifier=code_verifier) + assert token == {'access_token': token['access_token'], 'token_type': 'Bearer', 'scope': 'read:user'} + + with app.test_client() as test_client_no_session: + # make sure we can use our token + uri, headers, data = oauth_client.token_auth.prepare('/api/user/', {}, '') + resp = test_client_no_session.get(uri, data=data, headers=headers) + assert resp.status_code == 200 + assert resp.json['id'] == dummy_user.id + + # authorizing again won't require new consent, regardless of the app being trusted + assert test_client.get(auth_url).status_code == 302 + + +@pytest.mark.parametrize('pkce_enabled', (False, True)) +def test_pkce_disabled(dummy_application, test_client, dummy_user, pkce_enabled): + dummy_application.allow_pkce_flow = pkce_enabled + oauth_client = OAuth2Client(MockSession(test_client), + dummy_application.client_id, None, + code_challenge_method='S256', + token_endpoint=url_for('oauth.oauth_token', _external=True), + redirect_uri=dummy_application.default_redirect_uri) + + with test_client.session_transaction() as sess: + sess.set_session_user(dummy_user) + + code_verifier = generate_token(64) + auth_endpoint = url_for('oauth.oauth_authorize', _external=True) + auth_url = oauth_client.create_authorization_url(auth_endpoint, scope='read:legacy_api', + code_verifier=code_verifier)[0] + + authorized_resp = test_client.post(auth_url, data={'confirm': '1'}) + assert authorized_resp.status_code == 302 + target_url = authorized_resp.headers['Location'] + + if pkce_enabled: + token = oauth_client.fetch_token(authorization_response=target_url, code_verifier=code_verifier) + assert token.keys() == {'access_token', 'token_type', 'scope'} + else: + with pytest.raises(ValueError) as exc_info: + oauth_client.fetch_token(authorization_response=target_url, code_verifier=code_verifier) + assert 'invalid_client' in str(exc_info.value) + + +def test_no_implicit_flow(dummy_application, test_client, dummy_user): + oauth_client = OAuth2Client(None, + dummy_application.client_id, + None, + scope='read:user', + response_type='token', + token_endpoint=url_for('oauth.oauth_token', _external=True), + redirect_uri=dummy_application.default_redirect_uri) + + auth_url = oauth_client.create_authorization_url(url_for('oauth.oauth_authorize', _external=True))[0] + + with test_client.session_transaction() as sess: + sess.set_session_user(dummy_user) + + resp = test_client.get(auth_url) + assert b'unsupported_response_type' in resp.data + + +def test_no_querystring_tokens(dummy_user, dummy_token, test_client): + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {dummy_token._plaintext_token}'}) + assert resp.status_code == 200 + assert resp.json['id'] == dummy_user.id + resp = test_client.get(f'/api/user/?access_token={dummy_token._plaintext_token}') + assert resp.status_code == 200 + assert resp.json is None # the API returns json `null` if not authenticated + + +def test_oauth_scopes(create_application, test_client, dummy_user, app): + oauth_app = create_application(name='test', is_trusted=False, allowed_scopes=['read:user', 'read:legacy_api']) + oauth_client = OAuth2Client(MockSession(test_client), + oauth_app.client_id, oauth_app.client_secret, + token_endpoint=url_for('oauth.oauth_token', _external=True), + redirect_uri=oauth_app.default_redirect_uri) + + with test_client.session_transaction() as sess: + sess.set_session_user(dummy_user) + + auth_endpoint = url_for('oauth.oauth_authorize', _external=True) + auth_url = oauth_client.create_authorization_url(auth_endpoint, scope='read:legacy_api')[0] + assert not oauth_app.user_links.count() + + # get consent page + resp = test_client.get(auth_url) + assert resp.status_code == 200 + assert b'is requesting the following permissions' in resp.data + assert b'Legacy API (read only)' in resp.data + assert b'User information (read only)' not in resp.data + + # give consent + authorized_resp = test_client.post(auth_url, data={'confirm': '1'}) + assert authorized_resp.status_code == 302 + target_url = authorized_resp.headers['Location'] + assert not oauth_app.user_links.count() # giving consent does not create the user/app link yet + + # get a token and make sure it looks fine + token1 = oauth_client.fetch_token(authorization_response=target_url) + assert token1 == {'access_token': token1['access_token'], 'token_type': 'Bearer', 'scope': 'read:legacy_api'} + assert len(token1['access_token']) == 42 # longer would be fine but we don't expect this to change + app_link = oauth_app.user_links.one() + assert app_link.user == dummy_user + assert app_link.scopes == ['read:legacy_api'] + + # we cannot use a token with an invalid scope + with app.test_client() as test_client_no_session: + uri, headers, data = oauth_client.token_auth.prepare('/api/user/', {}, '') + resp = test_client_no_session.get(uri, data=data, headers=headers) + assert resp.status_code == 403 + + # request a different scope + auth_url = oauth_client.create_authorization_url(auth_endpoint, scope='read:user')[0] + authorized_resp = test_client.post(auth_url, data={'confirm': '1'}) + target_url = authorized_resp.headers['Location'] + token2 = oauth_client.fetch_token(authorization_response=target_url) + assert token2 == {'access_token': token2['access_token'], 'token_type': 'Bearer', 'scope': 'read:user'} + assert token2['access_token'] != token1['access_token'] + assert app_link.scopes == ['read:legacy_api', 'read:user'] + + # this token is already able to access the endpoint + with app.test_client() as test_client_no_session: + uri, headers, data = oauth_client.token_auth.prepare('/api/user/', {}, '') + resp = test_client_no_session.get(uri, data=data, headers=headers) + assert resp.status_code == 200 + assert resp.json['id'] == dummy_user.id + + # no scope specified, so we should get a token with all authorized scopes + auth_url = oauth_client.create_authorization_url(auth_endpoint)[0] + authorized_resp = test_client.post(auth_url, data={'confirm': '1'}) + target_url = authorized_resp.headers['Location'] + token3 = oauth_client.fetch_token(authorization_response=target_url) + assert set(token3.pop('scope').split()) == {'read:legacy_api', 'read:user'} + assert token3 == {'access_token': token3['access_token'], 'token_type': 'Bearer'} + assert token3['access_token'] != token2['access_token'] + assert app_link.scopes == ['read:legacy_api', 'read:user'] + + # and of course that token also has access to the endpoint + with app.test_client() as test_client_no_session: + uri, headers, data = oauth_client.token_auth.prepare('/api/user/', {}, '') + resp = test_client_no_session.get(uri, data=data, headers=headers) + assert resp.status_code == 200 + assert resp.json['id'] == dummy_user.id + + # reuse an existing scope + auth_url = oauth_client.create_authorization_url(auth_endpoint, scope='read:user')[0] + authorized_resp = test_client.post(auth_url, data={'confirm': '1'}) + target_url = authorized_resp.headers['Location'] + token4 = oauth_client.fetch_token(authorization_response=target_url) + assert token4 != token2 # can't reuse the old token since it's only stored as a hash + + +@pytest.mark.parametrize('endpoint_auth', ('client_secret_post', 'client_secret_basic')) +def test_introspection(dummy_application, dummy_token, test_client, endpoint_auth): + data = {'token': dummy_token._plaintext_token} + headers = {} + if endpoint_auth == 'client_secret_post': + data['client_id'] = dummy_application.client_id + data['client_secret'] = dummy_application.client_secret + elif endpoint_auth == 'client_secret_basic': + basic_auth = b64encode(f'{dummy_application.client_id}:{dummy_application.client_secret}'.encode()).decode() + headers['Authorization'] = f'Basic {basic_auth}' + resp = test_client.post('/oauth/introspect', data=data, headers=headers) + assert resp.json == { + 'active': True, + 'client_id': dummy_application.client_id, + 'token_type': 'Bearer', + 'scope': dummy_token.get_scope(), + 'sub': str(dummy_token.user.id), + 'iss': 'http://localhost' + } + + +@pytest.mark.parametrize('reason', ('nouuid', 'invalid', 'appdisabled')) +def test_introspection_inactive(dummy_application, dummy_token, test_client, reason): + token = dummy_token._plaintext_token + if reason == 'nouuid': + token = 'garbage' + elif reason == 'invalid': + token = '00000000-0000-0000-0000-000000000000' + elif reason == 'appdisabled': + dummy_application.is_enabled = False + + data = { + 'token': token, + 'client_id': dummy_application.client_id, + 'client_secret': dummy_application.client_secret + } + resp = test_client.post('/oauth/introspect', data=data) + if reason == 'appdisabled': + assert resp.status_code == 400 + assert resp.json == {'error': 'invalid_client'} + else: + assert resp.json == {'active': False} + + +def test_introspection_wrong_app(create_application, dummy_token, test_client): + other_app = create_application(name='test') + data = { + 'token': dummy_token._plaintext_token, + 'client_id': other_app.client_id, + 'client_secret': other_app.client_secret + } + resp = test_client.post('/oauth/introspect', data=data) + assert resp.json == {'active': False} + + +@pytest.mark.parametrize('endpoint_auth', ('client_secret_post', 'client_secret_basic')) +def test_revocation(db, dummy_application, dummy_token, test_client, endpoint_auth): + data = {'token': dummy_token._plaintext_token} + headers = {} + if endpoint_auth == 'client_secret_post': + data['client_id'] = dummy_application.client_id + data['client_secret'] = dummy_application.client_secret + elif endpoint_auth == 'client_secret_basic': + basic_auth = b64encode(f'{dummy_application.client_id}:{dummy_application.client_secret}'.encode()).decode() + headers['Authorization'] = f'Basic {basic_auth}' + resp = test_client.post('/oauth/revoke', data=data, headers=headers) + assert resp.status_code == 200 + assert resp.json == {} + assert dummy_token not in db.session + # make sure we can no longer use the token + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {dummy_token._plaintext_token}'}) + assert resp.status_code == 401 + + +def test_revocation_wrong_app(db, create_application, dummy_token, test_client): + other_app = create_application(name='test') + data = { + 'token': dummy_token._plaintext_token, + 'client_id': other_app.client_id, + 'client_secret': other_app.client_secret + } + resp = test_client.post('/oauth/revoke', data=data) + assert resp.status_code == 200 + assert resp.json == {} + assert dummy_token in db.session + # make sure we can still use the token + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {dummy_token._plaintext_token}'}) + assert resp.status_code == 200 + + +@pytest.mark.parametrize(('reason', 'status_code', 'error'), ( + ('nouuid', 401, 'invalid_token'), + ('invalid', 401, 'invalid_token'), + ('appdisabled', 401, 'invalid_token'), + ('badscope', 403, 'insufficient_scope'), + ('badapplinkscope', 403, 'insufficient_scope'), + ('badappscope', 403, 'insufficient_scope') +)) +def test_invalid_token(dummy_application, dummy_token, test_client, reason, status_code, error): + token = dummy_token._plaintext_token + if reason == 'nouuid': + token = 'garbage' + elif reason == 'invalid': + token = '00000000-0000-0000-0000-000000000000' + elif reason == 'appdisabled': + dummy_application.is_enabled = False + + if reason == 'badscope': + dummy_token._scopes.remove('read:user') + elif reason == 'badapplinkscope': + dummy_token.app_user_link.scopes.remove('read:user') + elif reason == 'badappscope': + dummy_application.allowed_scopes.remove('read:user') + + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {token}'}) + assert resp.status_code == status_code + assert resp.json['error'] == error + + +def test_metadata_endpoint(test_client): + resp = test_client.get('/.well-known/oauth-authorization-server') + assert resp.status_code == 200 + assert resp.json == { + 'authorization_endpoint': 'http://localhost/oauth/authorize', + 'code_challenge_methods_supported': ['S256'], + 'grant_types_supported': ['authorization_code'], + 'introspection_endpoint': 'http://localhost/oauth/introspect', + 'introspection_endpoint_auth_methods_supported': ['client_secret_basic', 'client_secret_post'], + 'issuer': 'http://localhost', + 'response_modes_supported': ['query'], + 'response_types_supported': ['code'], + 'revocation_endpoint': 'http://localhost/oauth/revoke', + 'revocation_endpoint_auth_methods_supported': ['client_secret_basic', 'client_secret_post'], + 'scopes_supported': list(SCOPES), + 'token_endpoint': 'http://localhost/oauth/token', + 'token_endpoint_auth_methods_supported': ['client_secret_basic', 'client_secret_post', 'none'] + } + + +def test_delete_old_tokens(db, dummy_application, dummy_user): + request = MagicMock(client=dummy_application, user=dummy_user) + + gen_hashes = {'foo': [], 'bar': []} + for scope in ('foo', 'bar'): + for i in range(MAX_TOKENS_PER_SCOPE + 2): + token_string = generate_token(69) + gen_hashes[scope].append(hashlib.sha256(token_string.encode()).hexdigest()) + save_token({'scope': scope, 'access_token': token_string}, request) + num_tokens = OAuthToken.query.filter_by(_scopes=db.cast([scope], ARRAY(db.String))).count() + assert num_tokens == min(i + 1, MAX_TOKENS_PER_SCOPE) + + # ensure we have the latest MAX_TOKENS_PER_SCOPE tokens in the DB + for scope in ('foo', 'bar'): + query = (db.session.query(OAuthToken.access_token_hash) + .filter_by(_scopes=db.cast([scope], ARRAY(db.String))) + .order_by(OAuthToken.created_dt)) + db_hashes = [x.access_token_hash for x in query] + assert db_hashes == gen_hashes[scope][-MAX_TOKENS_PER_SCOPE:] + + +def test_merge_users(create_user, dummy_user, dummy_application, dummy_token, create_application, test_client): + source_user = create_user(123) + + # app on both users (already exists on dummy user via dummy token) + app_link = OAuthApplicationUserLink(application=dummy_application, user=source_user, + scopes=['read:user', 'write:legacy_api']) + token_string = generate_token() + OAuthToken(access_token=token_string, app_user_link=app_link, scopes=['read:user']) + + # app only on source user + test_app = create_application(name='test') + app_link2 = OAuthApplicationUserLink(application=test_app, user=source_user, scopes=['read:user']) + token_string2 = generate_token() + OAuthToken(access_token=token_string2, app_user_link=app_link2, scopes=['read:user']) + OAuthToken(access_token=generate_token(), app_user_link=app_link2, scopes=['read:user']) + OAuthToken(access_token=generate_token(), app_user_link=app_link2, scopes=['read:user']) + + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {dummy_token._plaintext_token}'}) + assert resp.status_code == 200 + assert resp.json['id'] == dummy_user.id + + for token in (token_string, token_string2): + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {token}'}) + assert resp.status_code == 200 + assert resp.json['id'] == source_user.id + + old_token_count = OAuthToken.query.count() + merge_users(source_user, dummy_user) + + # source user should not have any leftover app links + assert not source_user.oauth_app_links.count() + # two app links on the target user + assert dummy_user.oauth_app_links.count() == 2 + # dummy app has one token from each user + assert dummy_user.oauth_app_links.filter_by(application=dummy_application).one().tokens.count() == 2 + # test app has 3 tokens coming from source user + assert dummy_user.oauth_app_links.filter_by(application=test_app).one().tokens.count() == 3 + # the total number of tokens didn't change (we do not delete surplus tokens during merge anyway) + assert OAuthToken.query.count() == old_token_count + + # all tokens point to the target user + for token in (dummy_token._plaintext_token, token_string, token_string2): + resp = test_client.get('/api/user/', headers={'Authorization': f'Bearer {token}'}) + assert resp.status_code == 200 + assert resp.json['id'] == dummy_user.id diff --git a/indico/core/oauth/protector.py b/indico/core/oauth/protector.py new file mode 100644 index 00000000000..9859f6f1c2a --- /dev/null +++ b/indico/core/oauth/protector.py @@ -0,0 +1,56 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from authlib.integrations.flask_oauth2 import ResourceProtector +from authlib.oauth2.rfc6750.validator import BearerTokenValidator +from flask import after_this_request, jsonify +from werkzeug.exceptions import HTTPException + +from indico.core.db import db +from indico.core.oauth.models.tokens import OAuthToken +from indico.core.oauth.util import query_token +from indico.util.date_time import now_utc + + +class IndicoAuthlibHTTPError(HTTPException): + def __init__(self, status_code, payload, headers): + super().__init__(payload.get('error_description') or payload['error']) + resp = jsonify(payload) + resp.headers.update(headers) + resp.status_code = status_code + self.response = resp + + +class IndicoResourceProtector(ResourceProtector): + def raise_error_response(self, error): + payload = dict(error.get_body()) + headers = error.get_headers() + raise IndicoAuthlibHTTPError(error.status_code, payload, headers) + + +class IndicoBearerTokenValidator(BearerTokenValidator): + def authenticate_token(self, token_string): + return query_token(token_string) + + def validate_token(self, token, scopes): + super().validate_token(token, scopes) + + # if we get here, the token is valid so we can mark it as used at the end of the request + + # XXX: should we wait or do it just now? even if the request failed for some reason, the + # token could be considered used, since it was valid and most likely used by a client who + # expected to do something with it... + + token_id = token.id # avoid DetachedInstanceError in the callback + + @after_this_request + def _update_last_use(response): + with db.tmp_session() as sess: + # do not modify `token` directly, it's attached to a different session! + sess.query(OAuthToken).filter_by(id=token_id).update({OAuthToken.last_used_dt: now_utc()}) + sess.commit() + return response diff --git a/indico/core/oauth/scopes.py b/indico/core/oauth/scopes.py new file mode 100644 index 00000000000..54f45e2eeaf --- /dev/null +++ b/indico/core/oauth/scopes.py @@ -0,0 +1,18 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from indico.util.i18n import _ + + +SCOPES = { + 'read:user': _("User information (read only)"), + 'read:legacy_api': _('Legacy API (read only)'), + 'write:legacy_api': _('Legacy API (write only)'), + 'registrants': _('Event registrants'), + 'read:everything': _('Everything (only GET)'), + 'full:everything': _('Everything (all methods)'), +} diff --git a/indico/legacy/services/__init__.py b/indico/core/oauth/testing/__init__.py similarity index 100% rename from indico/legacy/services/__init__.py rename to indico/core/oauth/testing/__init__.py diff --git a/indico/core/oauth/testing/fixtures.py b/indico/core/oauth/testing/fixtures.py new file mode 100644 index 00000000000..eca5ebdb95e --- /dev/null +++ b/indico/core/oauth/testing/fixtures.py @@ -0,0 +1,58 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from uuid import uuid4 + +import pytest +from authlib.common.security import generate_token + +from indico.core.oauth.models.applications import OAuthApplication, OAuthApplicationUserLink +from indico.core.oauth.models.tokens import OAuthToken + + +@pytest.fixture +def create_application(db): + """Return a callable which lets you create applications.""" + + def _create_application(name, **params): + params.setdefault('client_id', str(uuid4())) + params.setdefault('allowed_scopes', ['read:legacy_api', 'write:legacy_api', 'read:user']) + params.setdefault('redirect_uris', ['http://localhost:10500/']) + params.setdefault('allow_pkce_flow', True) + application = OAuthApplication(name=name, **params) + db.session.add(application) + db.session.flush() + return application + + return _create_application + + +@pytest.fixture +def dummy_application(create_application): + """Return a dummy application.""" + return create_application(name='dummy') + + +@pytest.fixture +def dummy_app_link(db, dummy_application, dummy_user): + """Return an app link for the dummy user.""" + link = OAuthApplicationUserLink(application=dummy_application, user=dummy_user, + scopes=['read:legacy_api', 'read:user']) + db.session.add(link) + db.session.flush() + return link + + +@pytest.fixture +def dummy_token(db, dummy_app_link): + """Return a token for the dummy app/user.""" + token_string = generate_token() + token = OAuthToken(access_token=token_string, app_user_link=dummy_app_link, scopes=['read:legacy_api', 'read:user']) + token._plaintext_token = token_string + db.session.add(token) + db.session.flush() + return token diff --git a/indico/core/oauth/util.py b/indico/core/oauth/util.py new file mode 100644 index 00000000000..eac877bedcb --- /dev/null +++ b/indico/core/oauth/util.py @@ -0,0 +1,70 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +import hashlib +from uuid import UUID + +from authlib.oauth2.rfc6749 import list_to_scope, scope_to_list +from sqlalchemy.dialects.postgresql.array import ARRAY +from sqlalchemy.orm import joinedload + +from indico.core.db import db +from indico.core.oauth.logger import logger +from indico.core.oauth.models.applications import OAuthApplication, OAuthApplicationUserLink +from indico.core.oauth.models.tokens import OAuthToken + + +# The maximum number of tokens to keep for any given app/user and scope combination +MAX_TOKENS_PER_SCOPE = 50 + + +def query_token(token_string): + token_hash = hashlib.sha256(token_string.encode()).hexdigest() + # we always need the app link (which already loads the application) and the user + # since we need those to check if the token is still valid + return (OAuthToken.query + .filter_by(access_token_hash=token_hash) + .options(joinedload('app_user_link').joinedload('user')) + .first()) + + +def query_client(client_id): + try: + UUID(hex=client_id) + except ValueError: + return None + return OAuthApplication.query.filter_by(client_id=client_id, is_enabled=True).first() + + +def save_token(token_data, request): + requested_scopes = set(scope_to_list(token_data.get('scope', ''))) + application = OAuthApplication.query.filter_by(client_id=request.client.client_id).one() + link = OAuthApplicationUserLink.query.with_parent(application).with_parent(request.user).first() + + if link is None: + link = OAuthApplicationUserLink(application=application, user=request.user, scopes=requested_scopes) + else: + if not requested_scopes: + # for already-authorized apps not specifying a scope uses all scopes the + # user previously granted to the app + requested_scopes = set(link.scopes) + token_data['scope'] = list_to_scope(requested_scopes) + new_scopes = requested_scopes - set(link.scopes) + if new_scopes: + logger.info('New scopes for %r: %s', link, new_scopes) + link.update_scopes(new_scopes) + + link.tokens.append(OAuthToken(access_token=token_data['access_token'], scopes=requested_scopes)) + + # get rid of old tokens if there are too many + q = (db.session.query(OAuthToken.id) + .with_parent(link) + .filter_by(_scopes=db.cast(sorted(requested_scopes), ARRAY(db.String))) + .order_by(OAuthToken.created_dt.desc()) + .offset(MAX_TOKENS_PER_SCOPE) + .scalar_subquery()) + OAuthToken.query.filter(OAuthToken.id.in_(q)).delete(synchronize_session='fetch') diff --git a/indico/core/permissions.py b/indico/core/permissions.py index b0292c87807..f133832d0a8 100644 --- a/indico/core/permissions.py +++ b/indico/core/permissions.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core import signals from indico.util.caching import memoize_request from indico.util.i18n import _ @@ -17,7 +15,7 @@ READ_ACCESS_PERMISSION = '_read_access' -class ManagementPermission(object): +class ManagementPermission: """Base class for management permissions. To create a new permission, subclass this class and register @@ -41,13 +39,13 @@ class ManagementPermission(object): @memoize_request def get_available_permissions(type_): - """Gets a dict containing all permissions for a given object type""" + """Get a dict containing all permissions for a given object type.""" return named_objects_from_signal(signals.acl.get_management_permissions.send(type_)) def check_permissions(type_): """ - Retrieves the permissions for an object type and ensures they are + Retrieve the permissions for an object type and ensure they are defined properly. This function should be executed from a function connected to the @@ -57,21 +55,24 @@ def check_permissions(type_): permissions = get_available_permissions(type_) if not all(x.islower() for x in permissions): raise RuntimeError('Management permissions must be all-lowercase') - if len(list(x for x in permissions.viewvalues() if x.default)) > 1: + if len(list(x for x in permissions.values() if x.default)) > 1: raise RuntimeError('Only one permission can be the default') def get_permissions_info(_type): - """Retrieve the permissions that can be set in the protection page and related information + """ + Retrieve the permissions that can be set in the protection page + and related information. + :param _type: The type of the permissions retrieved (e.g. Event, Category) :return: A tuple containing a dict with the available permissions and a dict with the permissions tree """ - from indico.modules.events import Event from indico.modules.categories import Category + from indico.modules.events import Event from indico.modules.events.contributions import Contribution from indico.modules.events.sessions import Session - from indico.modules.rb.models.rooms import Room from indico.modules.events.tracks import Track + from indico.modules.rb.models.rooms import Room description_mapping = { FULL_ACCESS_PERMISSION: { @@ -92,7 +93,7 @@ def get_permissions_info(_type): } } - selectable_permissions = {k: v for k, v in get_available_permissions(_type).viewitems() if v.user_selectable} + selectable_permissions = {k: v for k, v in get_available_permissions(_type).items() if v.user_selectable} special_permissions = { FULL_ACCESS_PERMISSION: { 'title': _('Manage'), @@ -115,7 +116,7 @@ def get_permissions_info(_type): 'description': special_permissions[FULL_ACCESS_PERMISSION]['description'], 'children': { perm.name: {'title': perm.friendly_name, 'description': perm.description} - for name, perm in selectable_permissions.viewitems() + for name, perm in selectable_permissions.items() } }, READ_ACCESS_PERMISSION: { @@ -125,12 +126,12 @@ def get_permissions_info(_type): } available_permissions = dict({k: { 'title': v.friendly_name, - 'css_class': 'permission-{}-{}'.format(_type.__name__.lower(), v.name), + 'css_class': f'permission-{_type.__name__.lower()}-{v.name}', 'description': v.description, 'default': v.default, 'color': v.color, - } for k, v in selectable_permissions.viewitems()}, **special_permissions) - default = next((k for k, v in available_permissions.viewitems() if v['default']), None) + } for k, v in selectable_permissions.items()}, **special_permissions) + default = next((k for k, v in available_permissions.items() if v['default']), None) return available_permissions, permissions_tree, default @@ -163,7 +164,10 @@ def get_unified_permissions(principal, all_permissions=False): def get_split_permissions(permissions): - """Split a list of permissions into a `has_full_access, has_read_access, list_with_others` tuple.""" + """ + Split a list of permissions into a `has_full_access, has_read_access, + list_with_others` tuple. + """ full_access_permission = FULL_ACCESS_PERMISSION in permissions read_access_permission = READ_ACCESS_PERMISSION in permissions other_permissions = permissions - {FULL_ACCESS_PERMISSION, READ_ACCESS_PERMISSION} @@ -172,31 +176,33 @@ def get_split_permissions(permissions): def update_permissions(obj, form): """Update the permissions of an object, based on the corresponding WTForm.""" - from indico.util.user import principal_from_fossil from indico.modules.categories import Category from indico.modules.events import Event + from indico.util.user import principal_from_identifier - event = category = None + event_id = category_id = None if isinstance(obj, Category): - category = obj + category_id = obj.id elif isinstance(obj, Event): - event = obj + event_id = obj.id else: - event = obj.event - category = event.category + event_id = obj.event.id + category_id = obj.event.category.id current_principal_permissions = {p.principal: get_principal_permissions(p, type(obj)) for p in obj.acl_entries} - current_principal_permissions = {k: v for k, v in current_principal_permissions.iteritems() if v} + current_principal_permissions = {k: v for k, v in current_principal_permissions.items() if v} new_principal_permissions = { - principal_from_fossil( - fossil, - allow_emails=True, + principal_from_identifier( + fossil['identifier'], + allow_groups=True, allow_networks=True, - allow_pending=True, - allow_registration_forms=True, - event=event, - category=category, + allow_emails=True, + allow_registration_forms=(event_id is not None), + allow_event_roles=(event_id is not None), + allow_category_roles=(event_id is not None or category_id is not None), + event_id=event_id, + category_id=category_id, ): set(permissions) for fossil, permissions in form.permissions.data } @@ -204,15 +210,16 @@ def update_permissions(obj, form): def update_principals_permissions(obj, current, new): - """Handle the updates of permissions and creations/deletions of acl principals. + """ + Handle the updates of permissions and creations/deletions of acl principals. :param obj: The object to update. Must have ``acl_entries`` :param current: A dict mapping principals to a set with its current permissions :param new: A dict mapping principals to a set with its new permissions """ - user_selectable_permissions = {v.name for k, v in get_available_permissions(type(obj)).viewitems() + user_selectable_permissions = {v.name for k, v in get_available_permissions(type(obj)).items() if v.user_selectable} - for principal, permissions in current.viewitems(): + for principal, permissions in current.items(): if principal not in new: permissions_kwargs = { 'full_access': False, diff --git a/indico/core/plugins/__init__.py b/indico/core/plugins/__init__.py index 680408546fa..d13281d10f3 100644 --- a/indico/core/plugins/__init__.py +++ b/indico/core/plugins/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import errno import json import os @@ -18,8 +16,7 @@ from werkzeug.utils import cached_property from indico.core import signals -from indico.core.db import db -from indico.core.db.sqlalchemy.util.models import import_all_models +from indico.core.db.sqlalchemy.util.models import get_all_models, import_all_models from indico.core.logger import Logger from indico.core.settings import SettingsProxy from indico.core.webpack import IndicoManifestLoader @@ -27,8 +24,8 @@ from indico.modules.events.static.util import RewrittenManifest from indico.modules.users import UserSettingsProxy from indico.util.decorators import cached_classproperty, classproperty +from indico.util.enum import IndicoEnum from indico.util.i18n import NullDomain, _ -from indico.util.struct.enum import IndicoEnum from indico.web.flask.templating import get_template_module, register_template_hook from indico.web.flask.util import url_for, url_rule_to_js from indico.web.flask.wrappers import IndicoBlueprint, IndicoBlueprintSetupState @@ -36,7 +33,7 @@ from indico.web.views import WPJinjaMixin -class PluginCategory(unicode, IndicoEnum): +class PluginCategory(str, IndicoEnum): search = _('Search') synchronization = _('Synchronization') payment = _('Payment') @@ -44,9 +41,12 @@ class PluginCategory(unicode, IndicoEnum): videoconference = _('Videoconference') other = _('Other') + def __str__(self): + return self.value + class IndicoPlugin(Plugin): - """Base class for an Indico plugin + """Base class for an Indico plugin. All your plugins need to inherit from this class. It extends the `Plugin` class from Flask-PluginEngine with useful indico-specific @@ -108,13 +108,13 @@ def init(self): self._import_models() def _import_models(self): - old_models = set(db.Model._decl_class_registry.items()) + old_models = get_all_models() import_all_models(self.package_name) - added_models = set(db.Model._decl_class_registry.items()) - old_models + added_models = get_all_models() - old_models # Ensure that only plugin schemas have been touched. It would be nice if we could actually # restrict a plugin to plugin_PLUGNNAME but since we load all models from the plugin's package # which could contain more than one plugin this is not easily possible. - for name, model in added_models: + for model in added_models: schema = model.__table__.schema # Allow models with non-plugin schema if they specify `polymorphic_identity` without a dedicated table if ('polymorphic_identity' in getattr(model, '__mapper_args__', ()) @@ -122,7 +122,7 @@ def _import_models(self): continue if not schema.startswith('plugin_'): raise Exception("Plugin '{}' added a model which is not in a plugin schema ('{}' in '{}')" - .format(self.name, name, schema)) + .format(self.name, model.__name__, schema)) def connect(self, signal, receiver, **connect_kwargs): connect_kwargs['weak'] = False @@ -131,29 +131,28 @@ def connect(self, signal, receiver, **connect_kwargs): signal.connect(func, **connect_kwargs) def get_blueprints(self): - """Return blueprints to be registered on the application + """Return blueprints to be registered on the application. A single blueprint can be returned directly, for multiple blueprint you need to yield them or return an iterable. """ - pass def get_vars_js(self): - """Return a dictionary with variables to be added to vars.js file""" + """Return a dictionary with variables to be added to vars.js file.""" return None @cached_property def translation_path(self): - """ - Return translation files to be used by the plugin. - By default, get /translations, unless it does not exist + """Return translation files to be used by the plugin. + + By default, get /translations, unless it does not exist. """ translations_path = os.path.join(self.root_path, 'translations') return translations_path if os.path.exists(translations_path) else None @cached_property def translation_domain(self): - """Return the domain for this plugin's translation_path""" + """Return the domain for this plugin's translation_path.""" path = self.translation_path return Domain(path) if path else NullDomain() @@ -161,7 +160,7 @@ def _get_manifest(self): try: loader = IndicoManifestLoader(custom=False) return loader.load(os.path.join(self.root_path, 'static', 'dist', 'manifest.json')) - except IOError as exc: + except OSError as exc: if exc.errno != errno.ENOENT: raise return None @@ -178,7 +177,7 @@ def manifest(self): return self._get_manifest() def inject_bundle(self, name, view_class=None, subclasses=True, condition=None): - """Injects an asset bundle into Indico's pages + """Inject an asset bundle into Indico's pages. :param name: Name of the bundle :param view_class: If a WP class is specified, only inject it into pages using that class @@ -189,7 +188,10 @@ def inject_bundle(self, name, view_class=None, subclasses=True, condition=None): def _do_inject(sender): if condition is None or condition(): - return self.manifest[name] + try: + return self.manifest[name] + except TypeError: + raise RuntimeError(f'Assets for plugin {self.name} have not been built') if view_class is None: self.connect(signals.plugin.inject_bundle, _do_inject) @@ -203,75 +205,78 @@ def _func(sender): self.connect(signals.plugin.inject_bundle, _func) def inject_vars_js(self): - """Returns a string that will define variables for the plugin in the vars.js file""" + """ + Return a string that will define variables for the plugin in + the vars.js file. + """ vars_js = self.get_vars_js() if vars_js: - return 'var {}Plugin = {};'.format(self.name.title(), json.dumps(vars_js)) + return f'var {self.name.title()}Plugin = {json.dumps(vars_js)};' def template_hook(self, name, receiver, priority=50, markup=True): - """Registers a function to be called when a template hook is invoked. + """Register a function to be called when a template hook is invoked. - For details see :func:`~indico.web.flask.templating.register_template_hook` + For details see :func:`~indico.web.flask.templating.register_template_hook`. """ register_template_hook(name, receiver, priority, markup, self) @classproperty @classmethod def logger(cls): - return Logger.get('plugin.{}'.format(cls.name)) + return Logger.get(f'plugin.{cls.name}') @cached_classproperty @classmethod def settings(cls): - """:class:`SettingsProxy` for the plugin's settings""" + """:class:`SettingsProxy` for the plugin's settings.""" if cls.name is None: raise RuntimeError('Plugin has not been loaded yet') instance = cls.instance with instance.plugin_context(): # in case the default settings come from a property - return SettingsProxy('plugin_{}'.format(cls.name), instance.default_settings, cls.strict_settings, + return SettingsProxy(f'plugin_{cls.name}', instance.default_settings, cls.strict_settings, acls=cls.acl_settings, converters=cls.settings_converters) @cached_classproperty @classmethod def event_settings(cls): - """:class:`EventSettingsProxy` for the plugin's event-specific settings""" + """:class:`EventSettingsProxy` for the plugin's event-specific settings.""" if cls.name is None: raise RuntimeError('Plugin has not been loaded yet') instance = cls.instance with instance.plugin_context(): # in case the default settings come from a property - return EventSettingsProxy('plugin_{}'.format(cls.name), instance.default_event_settings, + return EventSettingsProxy(f'plugin_{cls.name}', instance.default_event_settings, cls.strict_settings, acls=cls.acl_event_settings, converters=cls.event_settings_converters) @cached_classproperty @classmethod def user_settings(cls): - """:class:`UserSettingsProxy` for the plugin's user-specific settings""" + """:class:`UserSettingsProxy` for the plugin's user-specific settings.""" if cls.name is None: raise RuntimeError('Plugin has not been loaded yet') instance = cls.instance with instance.plugin_context(): # in case the default settings come from a property - return UserSettingsProxy('plugin_{}'.format(cls.name), instance.default_user_settings, + return UserSettingsProxy(f'plugin_{cls.name}', instance.default_user_settings, cls.strict_settings, converters=cls.user_settings_converters) def plugin_url_rule_to_js(endpoint): """Like :func:`~indico.web.flask.util.url_rule_to_js` but prepending plugin name prefix to the endpoint""" if '.' in endpoint[1:]: # 'foo' or '.foo' should not get the prefix - endpoint = 'plugin_{}'.format(endpoint) + endpoint = f'plugin_{endpoint}' return url_rule_to_js(endpoint) def url_for_plugin(endpoint, *targets, **values): """Like :func:`~indico.web.flask.util.url_for` but prepending ``'plugin_'`` to the blueprint name.""" if '.' in endpoint[1:]: # 'foo' or '.foo' should not get the prefix - endpoint = 'plugin_{}'.format(endpoint) + endpoint = f'plugin_{endpoint}' return url_for(endpoint, *targets, **values) def get_plugin_template_module(template_name, **context): """Like :func:`~indico.web.flask.templating.get_template_module`, but using plugin templates""" - template_name = '{}:{}'.format(current_plugin.name, template_name) + template_name = f'{current_plugin.name}:{template_name}' return get_template_module(template_name, **context) @@ -283,9 +288,9 @@ class IndicoPluginBlueprintSetupState(PluginBlueprintSetupStateMixin, IndicoBlue def add_url_rule(self, rule, endpoint=None, view_func=None, **options): if rule.startswith('/static'): with self._unprefixed(): - super(IndicoPluginBlueprintSetupState, self).add_url_rule(rule, endpoint, view_func, **options) + super().add_url_rule(rule, endpoint, view_func, **options) else: - super(IndicoPluginBlueprintSetupState, self).add_url_rule(rule, endpoint, view_func, **options) + super().add_url_rule(rule, endpoint, view_func, **options) class IndicoPluginBlueprint(PluginBlueprintMixin, IndicoBlueprint): diff --git a/indico/core/plugins/alembic/env.py b/indico/core/plugins/alembic/env.py index 70d1e97ac33..b7d5151a33f 100644 --- a/indico/core/plugins/alembic/env.py +++ b/indico/core/plugins/alembic/env.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -31,13 +31,12 @@ config.set_main_option('sqlalchemy.url', current_app.config.get('SQLALCHEMY_DATABASE_URI')) target_metadata = current_app.extensions['migrate'].db.metadata -plugin_schema = 'plugin_{}'.format(current_plugin.name) -version_table = 'alembic_version_plugin_{}'.format(current_plugin.name) +plugin_schema = f'plugin_{current_plugin.name}' +version_table = f'alembic_version_plugin_{current_plugin.name}' -def _include_symbol(tablename, schema): - # We only include tables in this plugin's schema in migrations - return schema == plugin_schema +def _include_object(object_, name, type_, reflected, compare_to): + return type_ != 'table' or object_.schema == plugin_schema def _render_item(type_, obj, autogen_context): @@ -59,11 +58,10 @@ def run_migrations_offline(): Calls to context.execute() here emit the given string to the script output. - """ url = config.get_main_option('sqlalchemy.url') context.configure(url=url, target_metadata=target_metadata, include_schemas=True, - include_symbol=_include_symbol, render_item=_render_item, version_table=version_table, + include_object=_include_object, render_item=_render_item, version_table=version_table, version_table_schema='public', template_args={'toplevel_code': set()}) with context.begin_transaction(): @@ -75,7 +73,6 @@ def run_migrations_online(): In this scenario we need to create an Engine and associate a connection with the context. - """ engine = engine_from_config(config.get_section(config.config_ini_section), prefix='sqlalchemy.', @@ -83,7 +80,7 @@ def run_migrations_online(): connection = engine.connect() context.configure(connection=connection, target_metadata=target_metadata, include_schemas=True, - include_symbol=_include_symbol, render_item=_render_item, version_table=version_table, + include_object=_include_object, render_item=_render_item, version_table=version_table, version_table_schema='public', template_args={'toplevel_code': set()}) try: diff --git a/indico/core/plugins/blueprint.py b/indico/core/plugins/blueprint.py index 44ced46bc69..7fbcb527df8 100644 --- a/indico/core/plugins/blueprint.py +++ b/indico/core/plugins/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.plugins.controllers import RHPluginDetails, RHPlugins from indico.web.flask.wrappers import IndicoBlueprint diff --git a/indico/core/plugins/controllers.py b/indico/core/plugins/controllers.py index 93672178c60..71cd4fa7aea 100644 --- a/indico/core/plugins/controllers.py +++ b/indico/core/plugins/controllers.py @@ -1,13 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - -from collections import OrderedDict, defaultdict +from collections import defaultdict from operator import attrgetter from flask import flash, request @@ -27,7 +25,7 @@ class RHPluginsBase(RHAdminBase): class RHPlugins(RHPluginsBase): def _process(self): - plugins = [p for p in plugin_engine.get_active_plugins().viewvalues()] + plugins = [p for p in plugin_engine.get_active_plugins().values()] categories = defaultdict(list) other = [] for plugin in plugins: @@ -40,9 +38,9 @@ def _process(self): # listed in the front for category in categories: categories[category].sort(key=attrgetter('configurable', 'title')) - ordered_categories = OrderedDict(sorted(categories.items())) + ordered_categories = dict(sorted(categories.items())) if other: - ordered_categories[PluginCategory.other] = other + ordered_categories[PluginCategory.other] = sorted(other, key=attrgetter('configurable', 'title')) return WPPlugins.render_template('index.html', categorized_plugins=ordered_categories) diff --git a/indico/core/plugins/templates/index.html b/indico/core/plugins/templates/index.html index 90139215b29..f58b9f3005d 100644 --- a/indico/core/plugins/templates/index.html +++ b/indico/core/plugins/templates/index.html @@ -5,7 +5,7 @@ {% endblock %} {% block content %} - {% for category, plugins in categorized_plugins.iteritems() %} + {% for category, plugins in categorized_plugins.items() %}

{{ category }}

@@ -16,7 +16,7 @@

{{ category }}

{%- else %} title="{% trans %}This plugin has no configurable options{% endtrans %}" {%- endif %}>
- {{ plugin.version }} + {{ plugin.version }}
{{ plugin.title }} diff --git a/indico/core/plugins/views.py b/indico/core/plugins/views.py index 62a9bfba6c1..a61a976b58b 100644 --- a/indico/core/plugins/views.py +++ b/indico/core/plugins/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.admin.views import WPAdmin diff --git a/indico/core/sentry.py b/indico/core/sentry.py new file mode 100644 index 00000000000..91011b2bf0e --- /dev/null +++ b/indico/core/sentry.py @@ -0,0 +1,95 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +import logging +import re + +import requests +import sentry_sdk +from flask import request +from pkg_resources import iter_entry_points +from sentry_sdk.integrations.flask import FlaskIntegration +from sentry_sdk.integrations.logging import LoggingIntegration, ignore_logger +from sentry_sdk.integrations.pure_eval import PureEvalIntegration +from sentry_sdk.integrations.redis import RedisIntegration +from werkzeug.urls import url_parse + +import indico +from indico.core.config import config +from indico.core.logger import Logger +from indico.util.i18n import set_best_lang + + +logger = Logger.get('sentry') + + +def init_sentry(app): + plugin_packages = {ep.module_name.split('.')[0] for ep in iter_entry_points('indico.plugins')} + ignore_logger('indico.flask') + sentry_sdk.init( + dsn=config.SENTRY_DSN, + release=indico.__version__, + send_default_pii=True, + attach_stacktrace=True, + in_app_include=({'indico'} | plugin_packages), + integrations=[ + PureEvalIntegration(), + RedisIntegration(), + FlaskIntegration(transaction_style='url'), + LoggingIntegration(event_level=getattr(logging, config.SENTRY_LOGGING_LEVEL)) + ], + _experiments={'record_sql_params': True} + ) + + app.before_request(_set_request_info) + + +def _set_request_info(): + sentry_sdk.set_extra('Endpoint', str(request.url_rule.endpoint) if request.url_rule else None) + sentry_sdk.set_extra('Request ID', request.id) + sentry_sdk.set_tag('locale', set_best_lang()) + + +def submit_user_feedback(error_data, email, comment): + if not config.SENTRY_DSN: + return + + # get rid of credentials or query string in case they are present in the DSN + dsn = re.sub(r':[^@/]+(?=@)', '', config.SENTRY_DSN) + url = url_parse(dsn) + dsn = str(url.replace(query='')) + verify = url.decode_query().get('ca_certs', True) + url = str(url.replace(path='/api/embed/error-page/', netloc=url._split_netloc()[1], query='')) + url = _resolve_redirects(url, verify) + user_data = error_data['request_info']['user'] or {'name': 'Anonymous', 'email': config.NO_REPLY_EMAIL} + try: + rv = requests.post( + url, + params={ + 'dsn': dsn, + 'eventId': error_data['sentry_event_id'] + }, + data={ + 'name': user_data['name'], + 'email': email or user_data['email'], + 'comments': comment + }, + headers={'Origin': config.BASE_URL}, + verify=verify, + ) + rv.raise_for_status() + except Exception: + # don't bother users if this fails! + logger.exception('Could not submit user feedback') + + +def _resolve_redirects(url, verify): + try: + return requests.head(url, allow_redirects=True, verify=verify).url + except Exception: + logger.exception('Could not resolve redirects') + return url diff --git a/indico/core/settings/__init__.py b/indico/core/settings/__init__.py index 2876d99369c..2c8f29523c7 100644 --- a/indico/core/settings/__init__.py +++ b/indico/core/settings/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.settings.proxy import (ACLProxyBase, AttributeProxyProperty, PrefixSettingsProxy, SettingProperty, SettingsProxy, SettingsProxyBase) diff --git a/indico/core/settings/converters.py b/indico/core/settings/converters.py index 96a27d8fc14..9b45b3e1c71 100644 --- a/indico/core/settings/converters.py +++ b/indico/core/settings/converters.py @@ -1,13 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - -from collections import OrderedDict from datetime import timedelta import dateutil.parser @@ -18,7 +15,7 @@ from indico.core.db import db -class SettingConverter(object): +class SettingConverter: """ Implement a custom conversion between Python types and JSON-serializable types. @@ -92,7 +89,7 @@ def __init__(self, model): @cached_property def model(self): - model = getattr(db.m, self._model) if isinstance(self._model, basestring) else self._model + model = getattr(db.m, self._model) if isinstance(self._model, str) else self._model assert len(inspect(model).primary_key) == 1 return model @@ -125,7 +122,7 @@ def __init__(self, model, collection_class=list): @cached_property def model(self): - if isinstance(self._model, basestring): + if isinstance(self._model, str): return getattr(db.m, self._model) return self._model @@ -145,13 +142,3 @@ def to_python(self, value): if not value: return [] return self.collection_class(self.model.query.filter(self.column.in_(value))) - - -class OrderedDictConverter(SettingConverter): - @staticmethod - def from_python(value): - return value.items() - - @staticmethod - def to_python(value): - return OrderedDict(value) diff --git a/indico/core/settings/models/base.py b/indico/core/settings/models/base.py index 4fff10a7f62..794a0b5f112 100644 --- a/indico/core/settings/models/base.py +++ b/indico/core/settings/models/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import defaultdict from enum import Enum @@ -24,8 +22,8 @@ def _coerce_value(value): return value -class SettingsBase(object): - """Base class for any kind of setting tables""" +class SettingsBase: + """Base class for any kind of setting tables.""" id = db.Column( db.Integer, @@ -52,13 +50,16 @@ def __auto_table_args(): def delete(cls, module, *names, **kwargs): if not names: return - cls.find(cls.name.in_(names), cls.module == module, **kwargs).delete(synchronize_session='fetch') + (cls.query + .filter(cls.name.in_(names), cls.module == module) + .filter_by(**kwargs) + .delete(synchronize_session='fetch')) db.session.flush() cls._clear_cache() @classmethod def delete_all(cls, module, **kwargs): - cls.find(module=module, **kwargs).delete() + cls.query.filter_by(module=module, **kwargs).delete() db.session.flush() cls._clear_cache() @@ -67,7 +68,7 @@ def _get_cache(cls, kwargs): if not has_request_context(): # disable the cache by always returning an empty one return defaultdict(dict), False - key = (cls, frozenset(kwargs.viewitems())) + key = (cls, frozenset(kwargs.items())) try: return g.global_settings_cache[key], True except AttributeError: @@ -86,7 +87,7 @@ def _clear_cache(): class JSONSettingsBase(SettingsBase): - """Base class for setting tables with a JSON value""" + """Base class for setting tables with a JSON value.""" __tablename__ = 'settings' @@ -97,11 +98,11 @@ class JSONSettingsBase(SettingsBase): @classmethod def get_setting(cls, module, name, **kwargs): - return cls.find_first(module=module, name=name, **kwargs) + return cls.query.filter_by(module=module, name=name, **kwargs).first() @classmethod def get_all_settings(cls, module, **kwargs): - return {s.name: s for s in cls.find(module=module, **kwargs)} + return {s.name: s for s in cls.query.filter_by(module=module, **kwargs)} @classmethod def get_all(cls, module, **kwargs): @@ -109,7 +110,7 @@ def get_all(cls, module, **kwargs): if hit: return cache[module] else: - for s in cls.find(**kwargs): + for s in cls.query.filter_by(**kwargs): cache[s.module][s.name] = s.value return cache[module] @@ -133,17 +134,17 @@ def set(cls, module, name, value, **kwargs): @classmethod def set_multi(cls, module, items, **kwargs): existing = cls.get_all_settings(module, **kwargs) - for name in items.viewkeys() - existing.viewkeys(): + for name in items.keys() - existing.keys(): setting = cls(module=module, name=name, value=_coerce_value(items[name]), **kwargs) db.session.add(setting) - for name in items.viewkeys() & existing.viewkeys(): + for name in items.keys() & existing.keys(): existing[name].value = _coerce_value(items[name]) db.session.flush() cls._clear_cache() class PrincipalSettingsBase(PrincipalMixin, SettingsBase): - """Base class for principal setting tables""" + """Base class for principal setting tables.""" __tablename__ = 'settings_principals' # Additional columns used to identitfy a setting (e.g. user/event id) @@ -157,13 +158,13 @@ def unique_columns(cls): @classmethod def get_all_acls(cls, module, **kwargs): rv = defaultdict(set) - for setting in cls.find(module=module, **kwargs): + for setting in cls.query.filter_by(module=module, **kwargs): rv[setting.name].add(setting.principal) return rv @classmethod def get_acl(cls, module, name, raw=False, **kwargs): - return {x if raw else x.principal for x in cls.find(module=module, name=name, **kwargs)} + return {x if raw else x.principal for x in cls.query.filter_by(module=module, name=name, **kwargs)} @classmethod def set_acl(cls, module, name, acl, **kwargs): @@ -178,7 +179,7 @@ def set_acl(cls, module, name, acl, **kwargs): @classmethod def set_acl_multi(cls, module, items, **kwargs): - for name, acl in items.iteritems(): + for name, acl in items.items(): cls.set_acl(module, name, acl, **kwargs) @classmethod @@ -197,7 +198,7 @@ def remove_principal(cls, module, name, principal, **kwargs): @classmethod def merge_users(cls, module, target, source): settings = [(setting.module, setting.name, {x: getattr(setting, x) for x in cls.extra_key_cols}) - for setting in cls.find(module=module, type=PrincipalType.user, user=source)] + for setting in cls.query.filter_by(module=module, type=PrincipalType.user, user=source)] for module, name, extra in settings: cls.remove_principal(module, name, source, **extra) cls.add_principal(module, name, target, **extra) diff --git a/indico/core/settings/models/settings.py b/indico/core/settings/models/settings.py index fc8ef76bd31..90426266a79 100644 --- a/indico/core/settings/models/settings.py +++ b/indico/core/settings/models/settings.py @@ -1,22 +1,19 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db.sqlalchemy import db from indico.core.db.sqlalchemy.util.models import auto_table_args from indico.core.settings.models.base import JSONSettingsBase, PrincipalSettingsBase from indico.util.decorators import strict_classproperty -from indico.util.string import return_ascii -class CoreSettingsMixin(object): +class CoreSettingsMixin: @strict_classproperty @staticmethod def __auto_table_args(): @@ -34,9 +31,8 @@ def __auto_table_args(): def __table_args__(cls): return auto_table_args(cls) - @return_ascii def __repr__(self): - return ''.format(self.module, self.name, self.value) + return f'' class SettingPrincipal(PrincipalSettingsBase, CoreSettingsMixin, db.Model): @@ -46,6 +42,5 @@ class SettingPrincipal(PrincipalSettingsBase, CoreSettingsMixin, db.Model): def __table_args__(cls): return auto_table_args(cls) - @return_ascii def __repr__(self): - return ''.format(self.module, self.name, self.principal) + return f'' diff --git a/indico/core/settings/models/settings_test.py b/indico/core/settings/models/settings_test.py index c6ccdd7709d..7c087c4dc7f 100644 --- a/indico/core/settings/models/settings_test.py +++ b/indico/core/settings/models/settings_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/settings/proxy.py b/indico/core/settings/proxy.py index 74998069a83..ed64baa0ef5 100644 --- a/indico/core/settings/proxy.py +++ b/indico/core/settings/proxy.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import defaultdict from functools import partial, update_wrapper from operator import attrgetter @@ -15,11 +13,10 @@ from indico.core.settings.models.settings import Setting, SettingPrincipal from indico.core.settings.util import get_all_settings, get_setting, get_setting_acl -from indico.util.string import return_ascii -class ACLProxyBase(object): - """Base Proxy class for ACL settings""" +class ACLProxyBase: + """Base Proxy class for ACL settings.""" def __init__(self, proxy): self.proxy = proxy @@ -36,8 +33,8 @@ def _flush_cache(self): self.proxy._flush_cache() -class SettingsProxyBase(object): - """Base proxy class to access settings for a certain module +class SettingsProxyBase: + """Base proxy class to access settings for a certain module. :param module: the module to use :param defaults: default values to use if there's nothing in the db @@ -64,20 +61,19 @@ def __init__(self, module, defaults=None, strict=True, acls=None, converters=Non raise ValueError('cannot use strict mode with no defaults') if acls and not self.acl_proxy_class: raise ValueError('this proxy does not support acl settings') - if acls and self.acl_names & self.defaults.viewkeys(): + if acls and self.acl_names & self.defaults.keys(): raise ValueError('acl settings cannot have a default value') - if acls and converters and acls & converters.viewkeys(): + if acls and converters and acls & converters.keys(): raise ValueError('acl settings cannot have custom converters') - @return_ascii def __repr__(self): if self._bound_args: - return '<{}({}, {})>'.format(type(self).__name__, self.module, self._bound_args) + return f'<{type(self).__name__}({self.module}, {self._bound_args})>' else: - return '<{}({})>'.format(type(self).__name__, self.module) + return f'<{type(self).__name__}({self.module})>' def bind(self, *args): - """Returns a version of this proxy that is bound to some arguments. + """Return a version of this proxy that is bound to some arguments. This is useful for specialized versions of the proxy such as EventSettingsProxy where one might want to provide an easy-to-use @@ -87,7 +83,6 @@ def bind(self, *args): :param args: The positional argument that are prepended to each function call. """ - self_type = type(self) bound = self_type(self.module, self.defaults, self.strict) bound._bound_args = args @@ -104,7 +99,7 @@ def _check_name(self, name, acl=False): strict = self.strict or acl # acl settings always use strict mode collection = self.acl_names if acl else self.defaults if strict and name not in collection: - raise ValueError('invalid setting: {}.{}'.format(self.module, name)) + raise ValueError(f'invalid setting: {self.module}.{name}') def _split_names(self, names): # Returns a ``(regular_names, acl_names)`` tuple @@ -158,10 +153,10 @@ def _cache(self): class ACLProxy(ACLProxyBase): - """Proxy class for core ACL settings""" + """Proxy class for core ACL settings.""" def get(self, name): - """Retrieves an ACL setting + """Retrieve an ACL setting. :param name: Setting name """ @@ -169,7 +164,7 @@ def get(self, name): return get_setting_acl(SettingPrincipal, self, name, self._cache) def set(self, name, acl): - """Replaces an ACL with a new one + """Replace an ACL with a new one. :param name: Setting name :param acl: A set containing principals (users/groups) @@ -179,7 +174,7 @@ def set(self, name, acl): self._flush_cache() def contains_user(self, name, user): - """Checks if a user is in an ACL. + """Check if a user is in an ACL. To pass this check, the user can either be in the ACL itself or in a group in the ACL. @@ -191,7 +186,7 @@ def contains_user(self, name, user): return any(user in principal for principal in iter_acl(self.get(name))) def add_principal(self, name, principal): - """Adds a principal to an ACL + """Add a principal to an ACL. :param name: Setting name :param principal: A :class:`.User` or a :class:`.GroupProxy` @@ -201,7 +196,7 @@ def add_principal(self, name, principal): self._flush_cache() def remove_principal(self, name, principal): - """Removes a principal from an ACL + """Remove a principal from an ACL. :param name: Setting name :param principal: A :class:`.User` or a :class:`.GroupProxy` @@ -211,18 +206,18 @@ def remove_principal(self, name, principal): self._flush_cache() def merge_users(self, target, source): - """Replaces all ACL user entries for `source` with `target`""" + """Replace all ACL user entries for `source` with `target`.""" SettingPrincipal.merge_users(self.module, target, source) self._flush_cache() class SettingsProxy(SettingsProxyBase): - """Proxy class to access settings for a certain module""" + """Proxy class to access settings for a certain module.""" acl_proxy_class = ACLProxy def get_all(self, no_defaults=False): - """Retrieves all settings, including ACLs + """Retrieve all settings, including ACLs. :param no_defaults: Only return existing settings and ignore defaults. :return: Dict containing the settings @@ -230,7 +225,7 @@ def get_all(self, no_defaults=False): return get_all_settings(Setting, SettingPrincipal, self, no_defaults) def get(self, name, default=SettingsProxyBase.default_sentinel): - """Retrieves the value of a single setting. + """Retrieve the value of a single setting. :param name: Setting name :param default: Default value in case the setting does not exist @@ -240,7 +235,7 @@ def get(self, name, default=SettingsProxyBase.default_sentinel): return get_setting(Setting, self, name, default, self._cache) def set(self, name, value): - """Sets a single setting. + """Set a single setting. :param name: Setting name :param value: Setting value; must be JSON-serializable @@ -250,18 +245,18 @@ def set(self, name, value): self._flush_cache() def set_multi(self, items): - """Sets multiple settings at once. + """Set multiple settings at once. :param items: Dict containing the new settings """ - items = {k: self._convert_from_python(k, v) for k, v in items.iteritems()} + items = {k: self._convert_from_python(k, v) for k, v in items.items()} self._split_call(items, lambda x: Setting.set_multi(self.module, x), lambda x: SettingPrincipal.set_acl_multi(self.module, x)) self._flush_cache() def delete(self, *names): - """Deletes settings. + """Delete settings. :param names: One or more names of settings to delete """ @@ -271,13 +266,13 @@ def delete(self, *names): self._flush_cache() def delete_all(self): - """Deletes all settings.""" + """Delete all settings.""" Setting.delete_all(self.module) SettingPrincipal.delete_all(self.module) self._flush_cache() -class SettingProperty(object): +class SettingProperty: """Expose a SettingsProxy value as a property. Override `attr` in a subclass for settings proxies that are tied @@ -319,7 +314,7 @@ def __delete__(self, obj): self.proxy.delete(*self._make_args(obj, self.name)) -class AttributeProxyProperty(object): +class AttributeProxyProperty: def __init__(self, attr): self.attr = attr @@ -335,8 +330,8 @@ def __delete__(self, obj): delattr(getattr(obj, obj.proxied_attr), self.attr) -class PrefixSettingsProxy(object): - """A SettingsProxy that exposes settings with prefixes +class PrefixSettingsProxy: + """A SettingsProxy that exposes settings with prefixes. This allows for simple form handling when a single form contains settings from more than one proxy. @@ -372,8 +367,8 @@ def _resolve_prefix(self, name): def get_all(self, no_defaults=False, arg=None): rv = {} - for prefix, proxy in self.mapping.iteritems(): - for key, value in self._call(proxy.get_all, arg, no_defaults=no_defaults).iteritems(): + for prefix, proxy in self.mapping.items(): + for key, value in self._call(proxy.get_all, arg, no_defaults=no_defaults).items(): rv[prefix + self.sep + key] = value return rv @@ -387,10 +382,10 @@ def set(self, name, value, arg=None): def set_multi(self, items, arg=None): by_proxy = defaultdict(dict) - for name, value in items.iteritems(): + for name, value in items.items(): proxy, local_name = self._resolve_prefix(name) by_proxy[proxy][local_name] = value - for proxy, local_items in by_proxy.iteritems(): + for proxy, local_items in by_proxy.items(): self._call(proxy.set_multi, arg, local_items) def delete(self, *names, **kwargs): @@ -400,9 +395,9 @@ def delete(self, *names, **kwargs): for name in names: proxy, local_name = self._resolve_prefix(name) by_proxy[proxy].append(local_name) - for proxy, local_names in by_proxy.iteritems(): + for proxy, local_names in by_proxy.items(): self._call(proxy.delete, arg, *local_names) def delete_all(self, arg=None): - for proxy in self.mapping.itervalues(): + for proxy in self.mapping.values(): self._call(proxy.delete_all, arg) diff --git a/indico/core/settings/proxy_test.py b/indico/core/settings/proxy_test.py index 92c89c3968b..98a9adc8c1f 100644 --- a/indico/core/settings/proxy_test.py +++ b/indico/core/settings/proxy_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/settings/util.py b/indico/core/settings/util.py index dbf1753bc67..8e3a0fa6160 100644 --- a/indico/core/settings/util.py +++ b/indico/core/settings/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from copy import copy @@ -14,33 +12,33 @@ def _get_cache_key(proxy, name, kwargs): - return type(proxy), proxy.module, name, frozenset(kwargs.viewitems()) + return type(proxy), proxy.module, name, frozenset(kwargs.items()) def _preload_settings(cls, proxy, cache, **kwargs): settings = cls.get_all(proxy.module, **kwargs) - for name, value in settings.iteritems(): + for name, value in settings.items(): cache_key = _get_cache_key(proxy, name, kwargs) cache[cache_key] = value # cache missing entries as not in db - for name in proxy.defaults.viewkeys() - settings.viewkeys(): + for name in proxy.defaults.keys() - settings.keys(): cache_key = _get_cache_key(proxy, name, kwargs) cache[cache_key] = _not_in_db return settings def get_all_settings(cls, acl_cls, proxy, no_defaults, **kwargs): - """Helper function for SettingsProxy.get_all""" + """Helper function for SettingsProxy.get_all.""" if no_defaults: rv = cls.get_all(proxy.module, **kwargs) if acl_cls and proxy.acl_names: rv.update(acl_cls.get_all_acls(proxy.module, **kwargs)) - return {k: proxy._convert_to_python(k, v) for k, v in rv.iteritems()} + return {k: proxy._convert_to_python(k, v) for k, v in rv.items()} settings = dict(proxy.defaults) if acl_cls and proxy.acl_names: settings.update({name: set() for name in proxy.acl_names}) settings.update({k: proxy._convert_to_python(k, v) - for k, v in cls.get_all(proxy.module, **kwargs).iteritems() + for k, v in cls.get_all(proxy.module, **kwargs).items() if not proxy.strict or k in proxy.defaults}) if acl_cls and proxy.acl_names: settings.update(acl_cls.get_all_acls(proxy.module, **kwargs)) @@ -48,7 +46,7 @@ def get_all_settings(cls, acl_cls, proxy, no_defaults, **kwargs): def get_setting(cls, proxy, name, default, cache, **kwargs): - """Helper function for SettingsProxy.get""" + """Helper function for SettingsProxy.get.""" from indico.core.settings import SettingsProxyBase cache_key = _get_cache_key(proxy, name, kwargs) @@ -67,7 +65,7 @@ def get_setting(cls, proxy, name, default, cache, **kwargs): def get_setting_acl(cls, proxy, name, cache, **kwargs): - """Helper function for ACLProxy.get""" + """Helper function for ACLProxy.get.""" cache_key = _get_cache_key(proxy, name, kwargs) try: return cache[cache_key] diff --git a/indico/core/signals/__init__.py b/indico/core/signals/__init__.py index d8ffeb17b7a..5adc3bd4c0b 100644 --- a/indico/core/signals/__init__.py +++ b/indico/core/signals/__init__.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/acl.py b/indico/core/signals/acl.py index 0ee604b6360..3449751ea21 100644 --- a/indico/core/signals/acl.py +++ b/indico/core/signals/acl.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/agreements.py b/indico/core/signals/agreements.py index 4ccd4481455..25a931006ce 100644 --- a/indico/core/signals/agreements.py +++ b/indico/core/signals/agreements.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/attachments.py b/indico/core/signals/attachments.py index 88ca8ee6142..57016a6be80 100644 --- a/indico/core/signals/attachments.py +++ b/indico/core/signals/attachments.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from blinker import Namespace diff --git a/indico/core/signals/category.py b/indico/core/signals/category.py index 5552b07f66f..7321b49ad2f 100644 --- a/indico/core/signals/category.py +++ b/indico/core/signals/category.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -21,7 +21,7 @@ Called when a new category is created. The `sender` is the new category. """) -updated = _signals.signal('created', """ +updated = _signals.signal('updated', """ Called when a category is modified. The `sender` is the updated category. """) diff --git a/indico/core/signals/core.py b/indico/core/signals/core.py index 99fd4b44224..aa9a796bd67 100644 --- a/indico/core/signals/core.py +++ b/indico/core/signals/core.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -86,3 +86,11 @@ Executed when a new database schema is created. The *sender* is the name of the schema. """) + +check_password_secure = _signals.signal('check-password-secure', """ +Check whether a password is secure. The *sender* is a string indicating +the context where the password check happens, the plaintext password is +sent in the *password* kwarg. To fail the security check for a password, +the signal handler should return a string describing why the password is +not secure. +""") diff --git a/indico/core/signals/event/__init__.py b/indico/core/signals/event/__init__.py index 0411069ca7a..01bd92131b0 100644 --- a/indico/core/signals/event/__init__.py +++ b/indico/core/signals/event/__init__.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/event/abstracts.py b/indico/core/signals/event/abstracts.py index 53869e865e7..ad5c87d9867 100644 --- a/indico/core/signals/event/abstracts.py +++ b/indico/core/signals/event/abstracts.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/event/contributions.py b/indico/core/signals/event/contributions.py index 03f11d58494..1e34a3f43cc 100644 --- a/indico/core/signals/event/contributions.py +++ b/indico/core/signals/event/contributions.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/event/core.py b/indico/core/signals/event/core.py index ccabcd5e568..027f4b69b74 100644 --- a/indico/core/signals/event/core.py +++ b/indico/core/signals/event/core.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -85,7 +85,8 @@ The *sender* is a string parameter specifying the source of the metadata. The *event* kwarg contains the event object. The metadata is passed in -the `data` kwarg. +the `data` kwarg. The `user` kwarg contains the user for whom the data is +generated. The signal should return a dict that will be used to update the original representation (fields to add or override). diff --git a/indico/core/signals/event/designer.py b/indico/core/signals/event/designer.py index 84b6b1882cd..3de4f115927 100644 --- a/indico/core/signals/event/designer.py +++ b/indico/core/signals/event/designer.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -12,6 +12,6 @@ print_badge_template = _signals.signal('print-badge-template', """ Called when printing a badge template. -The registration form is passed in the `regform` kwarg, the list of registration -objects are passed in the `registrations` kwarg. +The registration form is passed in the `regform` kwarg. The list of registration +objects are passed in the `registrations` kwarg and it may be modified. """) diff --git a/indico/core/signals/event/notes.py b/indico/core/signals/event/notes.py index 72e9d187a10..fd2b699119a 100644 --- a/indico/core/signals/event/notes.py +++ b/indico/core/signals/event/notes.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/event/persons.py b/indico/core/signals/event/persons.py index fc4685ce887..2df1c449d60 100644 --- a/indico/core/signals/event/persons.py +++ b/indico/core/signals/event/persons.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.signals.event import _signals diff --git a/indico/core/signals/event/registration.py b/indico/core/signals/event/registration.py index da4a3bf2d06..749e1ae1f00 100644 --- a/indico/core/signals/event/registration.py +++ b/indico/core/signals/event/registration.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from blinker import Namespace @@ -30,11 +28,13 @@ registration_created = _signals.signal('registration-created', """ Called when a new registration has been created. The `sender` is the `Registration` object. +The `data` kwarg contains the form data used to populate the registration fields. The `management` kwarg is set to `True` if the registration was created from the event management area. """) registration_updated = _signals.signal('registration-updated', """ Called when a registration has been updated. The `sender` is the `Registration` object. +The `data` kwarg contains the form data used to populate the registration fields. The `management` kwarg is set to `True` if the registration was updated from the event management area. """) @@ -42,6 +42,15 @@ Called when a registration is removed. The `sender` is the `Registration` object. """) +registration_form_wtform_created = _signals.signal('registration_form_wtform_created', """ +Called when a the wtform is created for rendering/processing a registration form. +The sender is the `RegistrationForm` object. The generated WTForm class is +passed in the `wtform_cls` kwarg and it may be modified. The `registration` +kwarg contains a `Registration` object when called from registration edit +endpoints. The `management` kwarg is set to `True` if the registration form is +rendered/processed from the event management area. +""") + registration_form_created = _signals.signal('registration-form-created', """ Called when a new registration form is created. The `sender` is the `RegistrationForm` object. diff --git a/indico/core/signals/event/timetable.py b/indico/core/signals/event/timetable.py index 6867b311538..df44ecd45e2 100644 --- a/indico/core/signals/event/timetable.py +++ b/indico/core/signals/event/timetable.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/event_management.py b/indico/core/signals/event_management.py index f8af7dd6d4c..1b1cc8a9ca8 100644 --- a/indico/core/signals/event_management.py +++ b/indico/core/signals/event_management.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/menu.py b/indico/core/signals/menu.py index 04f42271455..889a2867d7e 100644 --- a/indico/core/signals/menu.py +++ b/indico/core/signals/menu.py @@ -1,11 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. - from blinker import Namespace diff --git a/indico/core/signals/plugin.py b/indico/core/signals/plugin.py index ed7e4ee6cde..3df71dba1fa 100644 --- a/indico/core/signals/plugin.py +++ b/indico/core/signals/plugin.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/signals/rb.py b/indico/core/signals/rb.py index 88869783fbb..16ddf48bd8d 100644 --- a/indico/core/signals/rb.py +++ b/indico/core/signals/rb.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from blinker import Namespace diff --git a/indico/core/signals/rh.py b/indico/core/signals/rh.py index eefa7852144..2d3fb2b2fe4 100644 --- a/indico/core/signals/rh.py +++ b/indico/core/signals/rh.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from blinker import Namespace diff --git a/indico/core/signals/users.py b/indico/core/signals/users.py index 161a8b58b97..e061a4d5bb7 100644 --- a/indico/core/signals/users.py +++ b/indico/core/signals/users.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from blinker import Namespace @@ -21,6 +19,13 @@ the identity associated with the registration is passed in the `identity` kwarg. """) +logged_in = _signals.signal('logged-in', """ +Called when a user logs in. The *sender* is the User who logged in. Depending +on whether this was a regular login or an admin impersonating the user, either +the *identity* kwarg is set to the `Identity` used by the user to log in or the +*admin_impersonation* kwarg is ``True``. +""") + registration_requested = _signals.signal('registration-requested', """ Called when a user requests to register a new indico account, i.e. if moderation is enabled. The *sender* is the registration request. @@ -42,3 +47,8 @@ preferences page is being shown which might not be the currently logged-in user! """) + +primary_email_changed = _signals.signal('primary-email-changed', """ +Called when the primary address is changed. The *sender* is +the user object and the `new` and `old` values are passed as kwargs. +""") diff --git a/indico/core/storage/__init__.py b/indico/core/storage/__init__.py index 9c43d2fa297..8af8262dd94 100644 --- a/indico/core/storage/__init__.py +++ b/indico/core/storage/__init__.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/core/storage/backend.py b/indico/core/storage/backend.py index edf1c01fb05..481590d2d7d 100644 --- a/indico/core/storage/backend.py +++ b/indico/core/storage/backend.py @@ -1,14 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os -import sys from contextlib import contextmanager from hashlib import md5 from io import BytesIO @@ -19,12 +16,11 @@ from indico.core import signals from indico.core.config import config from indico.util.signals import named_objects_from_signal -from indico.util.string import return_ascii from indico.web.flask.util import send_file def get_storage(backend_name): - """Returns an FS object for the given backend. + """Return an FS object for the given backend. The backend must be defined in the STORAGE_BACKENDS dict in the indico config. Once a backend has been used it is assumed to @@ -37,12 +33,12 @@ def get_storage(backend_name): try: definition = config.STORAGE_BACKENDS[backend_name] except KeyError: - raise RuntimeError('Storage backend does not exist: {}'.format(backend_name)) + raise RuntimeError(f'Storage backend does not exist: {backend_name}') name, data = definition.split(':', 1) try: backend = get_storage_backends()[name] except KeyError: - raise RuntimeError('Storage backend {} has invalid type {}'.format(backend_name, name)) + raise RuntimeError(f'Storage backend {backend_name} has invalid type {name}') return backend(data) @@ -51,15 +47,15 @@ def get_storage_backends(): class StorageError(Exception): - """Exception used when a storage operation fails for any reason""" + """Exception used when a storage operation fails for any reason.""" class StorageReadOnlyError(StorageError): - """Exception used when trying to write to a read-only storage""" + """Exception used when trying to write to a read-only storage.""" -class Storage(object): - """Base class for storage backends +class Storage: + """Base class for storage backends. To create a new storage backend, subclass this class and register it using the `get_storage_backends` signal. @@ -79,6 +75,7 @@ class Storage(object): key-value pairs: ``key=value,key2=value2,..`` """ + #: unique name of the storage backend name = None #: plugin containing this backend - assigned automatically @@ -90,11 +87,11 @@ def __init__(self, data): # pragma: no cover pass def _parse_data(self, data): - """Util to parse a key=value data string to a dict""" + """Util to parse a key=value data string to a dict.""" return dict((x.strip() for x in item.split('=', 1)) for item in data.split(',')) if data else {} def _ensure_fileobj(self, fileobj): - """Ensures that fileobj is a file-like object and not a string""" + """Ensure that fileobj is a file-like object and not a string.""" return BytesIO(fileobj) if not hasattr(fileobj, 'read') else fileobj def _copy_file(self, source, target, chunk_size=1024*1024): @@ -109,10 +106,10 @@ def _copy_file(self, source, target, chunk_size=1024*1024): break target.write(chunk) checksum.update(chunk) - return checksum.hexdigest().decode('ascii') + return checksum.hexdigest() def open(self, file_id): # pragma: no cover - """Opens a file in the storage for reading. + """Open a file in the storage for reading. This returns a file-like object which contains the content of the file. @@ -123,7 +120,7 @@ def open(self, file_id): # pragma: no cover @contextmanager def get_local_path(self, file_id): - """Returns a local path for the file. + """Return a local path for the file. While this path MAY point to the permanent location of the stored file, it MUST NOT be used for anything but read @@ -139,7 +136,7 @@ def get_local_path(self, file_id): yield tmpfile.name def save(self, name, content_type, filename, fileobj): # pragma: no cover - """Creates a new file in the storage. + """Create a new file in the storage. This returns a a string identifier which can be used later to retrieve the file from the storage. @@ -168,21 +165,21 @@ def save(self, name, content_type, filename, fileobj): # pragma: no cover raise NotImplementedError def delete(self, file_id): # pragma: no cover - """Deletes a file from the storage. + """Delete a file from the storage. :param file_id: The ID of the file within the storage backend. """ raise NotImplementedError def getsize(self, file_id): # pragma: no cover - """Gets the size in bytes of a file + """Get the size in bytes of a file. :param file_id: The ID of the file within the storage backend. """ raise NotImplementedError def send_file(self, file_id, content_type, filename, inline=True): # pragma: no cover - """Sends the file to the client. + """Send the file to the client. This returns a flask response that will eventually result in the user being offered to download the file (or view it in the @@ -203,11 +200,11 @@ def send_file(self, file_id, content_type, filename, inline=True): # pragma: no raise NotImplementedError def __repr__(self): - return '<{}()>'.format(type(self).__name__) + return f'<{type(self).__name__}()>' -class ReadOnlyStorageMixin(object): - """Mixin that makes write operations fail with an error""" +class ReadOnlyStorageMixin: + """Mixin that makes write operations fail with an error.""" def save(self, name, content_type, filename, fileobj): raise StorageReadOnlyError('Cannot write to read-only storage') @@ -226,14 +223,14 @@ def __init__(self, data): def _resolve_path(self, path): full_path = safe_join(self.path, path) if full_path is None: - raise ValueError('Invalid path: {}'.format(path)) + raise ValueError(f'Invalid path: {path}') return full_path def open(self, file_id): try: return open(self._resolve_path(file_id), 'rb') except Exception as e: - raise StorageError('Could not open "{}": {}'.format(file_id, e)), None, sys.exc_info()[2] + raise StorageError(f'Could not open "{file_id}": {e}') from e @contextmanager def get_local_path(self, file_id): @@ -252,37 +249,35 @@ def save(self, name, content_type, filename, fileobj): checksum = self._copy_file(fileobj, f) return name, checksum except Exception as e: - raise StorageError('Could not save "{}": {}'.format(name, e)), None, sys.exc_info()[2] + raise StorageError(f'Could not save "{name}": {e}') from e def delete(self, file_id): try: os.remove(self._resolve_path(file_id)) except Exception as e: - raise StorageError('Could not delete "{}": {}'.format(file_id, e)), None, sys.exc_info()[2] + raise StorageError(f'Could not delete "{file_id}": {e}') from e def getsize(self, file_id): try: return os.path.getsize(self._resolve_path(file_id)) except Exception as e: - raise StorageError('Could not get size of "{}": {}'.format(file_id, e)), None, sys.exc_info()[2] + raise StorageError(f'Could not get size of "{file_id}": {e}') from e def send_file(self, file_id, content_type, filename, inline=True): try: - return send_file(filename, self._resolve_path(file_id).encode('utf-8'), content_type, inline=inline) + return send_file(filename, self._resolve_path(file_id), content_type, inline=inline) except Exception as e: - raise StorageError('Could not send "{}": {}'.format(file_id, e)), None, sys.exc_info()[2] + raise StorageError(f'Could not send "{file_id}": {e}') from e - @return_ascii def __repr__(self): - return ''.format(self.path) + return f'' class ReadOnlyFileSystemStorage(ReadOnlyStorageMixin, FileSystemStorage): name = 'fs-readonly' - @return_ascii def __repr__(self): - return ''.format(self.path) + return f'' @signals.get_storage_backends.connect diff --git a/indico/core/storage/backend_test.py b/indico/core/storage/backend_test.py index 3b4d1c1cff5..1e70a984c32 100644 --- a/indico/core/storage/backend_test.py +++ b/indico/core/storage/backend_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os from io import BytesIO @@ -41,23 +39,23 @@ def test_parse_data(data, expected): def test_fs_errors(fs_storage): with pytest.raises(StorageError) as exc_info: fs_storage.open('xxx') - assert 'Could not open' in unicode(exc_info.value) + assert 'Could not open' in str(exc_info.value) with pytest.raises(StorageError) as exc_info: fs_storage.send_file('xxx', 'unused/unused', 'unused') - assert 'Could not send' in unicode(exc_info.value) + assert 'Could not send' in str(exc_info.value) with pytest.raises(StorageError) as exc_info: fs_storage.delete('xxx') - assert 'Could not delete' in unicode(exc_info.value) + assert 'Could not delete' in str(exc_info.value) with pytest.raises(StorageError) as exc_info: fs_storage.getsize('xxx') - assert 'Could not get size' in unicode(exc_info.value) + assert 'Could not get size' in str(exc_info.value) with pytest.raises(StorageError) as exc_info: fs_storage.open('../xxx') - assert 'Invalid path' in unicode(exc_info.value) + assert 'Invalid path' in str(exc_info.value) os.mkdir(fs_storage._resolve_path('secret'), 0o000) with pytest.raises(StorageError) as exc_info: fs_storage.save('secret/test.txt', 'unused/unused', 'unused', b'hello test') - assert 'Could not save' in unicode(exc_info.value) + assert 'Could not save' in str(exc_info.value) os.rmdir(fs_storage._resolve_path('secret')) @@ -75,7 +73,7 @@ def test_fs_overwrite(fs_storage): f, __ = fs_storage.save('test.txt', 'unused/unused', 'unused', b'hello test') with pytest.raises(StorageError) as exc_info: fs_storage.save('test.txt', 'unused/unused', 'unused', b'hello fail') - assert 'already exists' in unicode(exc_info.value) + assert 'already exists' in str(exc_info.value) with fs_storage.open(f) as fd: assert fd.read() == b'hello test' @@ -85,11 +83,11 @@ def test_fs_dir(fs_storage): # Cannot open directory with pytest.raises(StorageError) as exc_info: fs_storage.open('foo') - assert 'Could not open' in unicode(exc_info.value) + assert 'Could not open' in str(exc_info.value) # Cannot create file colliding with the directory with pytest.raises(StorageError) as exc_info: fs_storage.save('foo', 'unused/unused', 'unused', b'hello test') - assert 'Could not save' in unicode(exc_info.value) + assert 'Could not save' in str(exc_info.value) def test_fs_operations(fs_storage): @@ -98,9 +96,9 @@ def test_fs_operations(fs_storage): f3, h3 = fs_storage.save('test.txt', 'unused/unused', 'unused', b'very very long file' * 1024 * 1024) # check md5 checksums - assert h1 == u'5eb63bbbe01eeed093cb22bb8f5acdc3' - assert h2 == u'161bc25962da8fed6d2f59922fb642aa' - assert h3 == u'd35ddfd803cbe8915f5c3ecd1d0523b4' + assert h1 == '5eb63bbbe01eeed093cb22bb8f5acdc3' + assert h2 == '161bc25962da8fed6d2f59922fb642aa' + assert h3 == 'd35ddfd803cbe8915f5c3ecd1d0523b4' with fs_storage.open(f1) as fd: assert fd.read() == b'hello world' @@ -125,7 +123,7 @@ def test_fs_send_file(fs_storage): response = fs_storage.send_file(f1, 'text/plain', 'filename.txt') assert 'text/plain' in response.headers['Content-type'] assert 'filename.txt' in response.headers['Content-disposition'] - assert ''.join(response.response) == 'hello world' + assert b''.join(response.response) == b'hello world' @pytest.mark.usefixtures('request_context') diff --git a/indico/core/storage/models.py b/indico/core/storage/models.py index fe84a7bca3c..8dffde656cb 100644 --- a/indico/core/storage/models.py +++ b/indico/core/storage/models.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -16,9 +16,6 @@ specifying, among others, which storage backend to use. """ - -from __future__ import unicode_literals - from sqlalchemy.event import listen from sqlalchemy.ext.declarative import declared_attr from sqlalchemy.orm import column_property @@ -29,7 +26,7 @@ from indico.util.date_time import now_utc -class VersionedResourceMixin(object): +class VersionedResourceMixin: # Class that will inherit from StoredFileMixin stored_file_class = None @@ -47,8 +44,10 @@ def _add_file_to_relationship(target, value, *unused): @classmethod def register_versioned_resource_events(cls): - """Register SQLAlchemy events. Should be called - right after class definition.""" + """Register SQLAlchemy events. + + Should be called right after class definition. + """ listen(cls.file, 'set', cls._add_file_to_relationship) #: The ID of the latest file for a file resource @@ -88,7 +87,7 @@ def all_files(cls): ) -class StoredFileMixin(object): +class StoredFileMixin: #: Name of attribute (backref) that will be made to point #: to the versioned resource (leave as ``None`` if you #: don't want versioning) @@ -101,7 +100,7 @@ class StoredFileMixin(object): @declared_attr def filename(cls): - """The name of the file""" + """The name of the file.""" return db.Column( db.String, nullable=not cls.file_required @@ -109,12 +108,12 @@ def filename(cls): @declared_attr def extension(cls): - """The extension of the file""" + """The extension of the file.""" return column_property(db.func.regexp_replace(cls.filename, r'^.*\.', ''), deferred=True) @declared_attr def content_type(cls): - """The MIME type of the file""" + """The MIME type of the file.""" return db.Column( db.String, nullable=not cls.file_required @@ -122,8 +121,7 @@ def content_type(cls): @declared_attr def size(cls): - """ - The size of the file (in bytes). + """The size of the file (in bytes). Automatically assigned when `save()` is called. """ @@ -134,8 +132,7 @@ def size(cls): @declared_attr def md5(cls): - """ - An MD5 hash of the file. + """An MD5 hash of the file. Automatically assigned when `save()` is called. """ @@ -160,7 +157,7 @@ def storage_file_id(cls): @declared_attr def created_dt(cls): - """The date/time when the file was uploaded""" + """The date/time when the file was uploaded.""" if not cls.add_file_date_column: return None return db.Column( @@ -178,17 +175,21 @@ def storage(self): def get_local_path(self): """Return context manager that will yield physical path. - This should be avoided in favour of using the actual file contents""" + + This should be avoided in favour of using the actual file contents. + """ return self.storage.get_local_path(self.storage_file_id) def _build_storage_path(self): - """Should return a tuple containing the name of the storage backend + """ + Should return a tuple containing the name of the storage backend to use and the actual path of that will be used to store the resource - using the former.""" + using the former. + """ raise NotImplementedError def save(self, data): - """Saves a file in the file storage. + """Save a file in the file storage. This requires the AttachmentFile to be associated with an Attachment which needs to be associated with a Folder since @@ -205,27 +206,30 @@ def save(self, data): self.size = self.storage.getsize(self.storage_file_id) def open(self): - """Returns the stored file as a file-like object""" + """Return the stored file as a file-like object.""" if self.storage_file_id is None: raise Exception('There is no file to open') return self.storage.open(self.storage_file_id) def send(self, inline=True): - """Sends the file to the user""" + """Send the file to the user.""" if self.storage_file_id is None: raise Exception('There is no file to send') return self.storage.send_file(self.storage_file_id, self.content_type, self.filename, inline=inline) def delete(self, delete_from_db=False): - """Delete the file from storage""" + """Delete the file from storage.""" if self.storage_file_id is None: raise Exception('There is no file to delete') - self.storage.delete(self.storage_file_id) + storage = self.storage + storage_file_id = self.storage_file_id if delete_from_db: db.session.delete(self) + db.session.flush() else: self.storage_backend = None self.storage_file_id = None self.size = None self.content_type = None self.filename = None + storage.delete(storage_file_id) diff --git a/indico/core/webpack.py b/indico/core/webpack.py index ff15be2bd57..68917cfad37 100644 --- a/indico/core/webpack.py +++ b/indico/core/webpack.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os from flask_webpackext import FlaskWebpackExt @@ -21,7 +19,7 @@ class IndicoManifestLoader(JinjaManifestLoader): def __init__(self, *args, **kwargs): self.custom = kwargs.pop('custom', True) - super(IndicoManifestLoader, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def load(self, filepath): key = (filepath, os.path.getmtime(filepath)) diff --git a/indico/legacy/common/cache.py b/indico/legacy/common/cache.py deleted file mode 100644 index f34d9bebf49..00000000000 --- a/indico/legacy/common/cache.py +++ /dev/null @@ -1,352 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -import cPickle as pickle -import datetime -import hashlib -import os -import time -from itertools import izip - -import redis -from flask import g - -from indico.core.config import config -from indico.core.logger import Logger -from indico.legacy.common.utils import OSSpecific -from indico.util.fs import silentremove -from indico.util.string import truncate - - -# To cache `None` we need to actually store something else since memcached -# does not distinguish between a None value and a cache miss... -class _NoneValue(object): - @classmethod - def replace(cls, value): - """Replaces `None` with a `_NoneValue`""" - return cls() if value is None else value - - @classmethod - def restore(cls, value): - """Replaces `_NoneValue` with `None`""" - return None if isinstance(value, cls) else value - - -class CacheClient(object): - """This is an abstract class. A cache client provide a simple API to get/set/delete cache entries. - - Implementation must provide the following methods: - - set(self, key, val, ttl) - - get(self, key) - - delete(self, key) - - The unit for the ttl arguments is a second. - """ - def set_multi(self, mapping, ttl=0): - for key, val in mapping.iteritems(): - self.set(key, val, ttl) - - def get_multi(self, keys): - values = {} - for key in keys: - val = self.get(key) - if val is not None: - values[key] = val - return values - - def delete_multi(self, keys): - for key in keys: - self.delete(key) - - def set(self, key, val, ttl=0): - raise NotImplementedError - - def get(self, key): - raise NotImplementedError - - def delete(self, key): - raise NotImplementedError - - -class NullCacheClient(CacheClient): - """Does nothing""" - - def set(self, key, val, ttl=0): - pass - - def get(self, key): - return None - - def delete(self, key): - pass - - -class RedisCacheClient(CacheClient): - """Redis-based cache client with a simple API""" - - key_prefix = 'cache/gen/' - - def __init__(self, url): - self._client = redis.StrictRedis.from_url(url) - self._client.connection_pool.connection_kwargs['socket_timeout'] = 1 - - def _unpickle(self, val): - if val is None: - return None - return pickle.loads(val) - - def hash_key(self, key): - # Redis keys are even binary-safe, no need to hash anything - return key - - def set_multi(self, mapping, ttl=0): - try: - self._client.mset(dict((k, pickle.dumps(v)) for k, v in mapping.iteritems())) - if ttl: - for key in mapping: - self._client.expire(key, ttl) - except redis.RedisError: - Logger.get('cache.redis').exception('set_multi(%r, %r) failed', mapping, ttl) - - def get_multi(self, keys): - try: - return dict(zip(keys, map(self._unpickle, self._client.mget(keys)))) - except redis.RedisError: - Logger.get('cache.redis').exception('get_multi(%r) failed', keys) - - def delete_multi(self, keys): - try: - self._client.delete(*keys) - except redis.RedisError: - Logger.get('cache.redis').exception('delete_multi(%r) failed', keys) - - def set(self, key, val, ttl=0): - try: - if ttl: - self._client.setex(key, ttl, pickle.dumps(val)) - else: - self._client.set(key, pickle.dumps(val)) - except redis.RedisError: - val_repr = truncate(repr(val), 1000) - Logger.get('cache.redis').exception('set(%r, %s, %r) failed', key, val_repr, ttl) - - def get(self, key): - try: - return self._unpickle(self._client.get(key)) - except redis.RedisError: - Logger.get('cache.redis').exception('get(%r) failed', key) - - def delete(self, key): - try: - self._client.delete(key) - except redis.RedisError: - Logger.get('cache.redis').exception('delete(%r) failed', key) - - -class FileCacheClient(CacheClient): - """File-based cache with a memcached-like API. - - Contains only features needed by GenericCache. - """ - def __init__(self, dir): - self._dir = os.path.join(dir, 'generic_cache') - - def _getFilePath(self, key, mkdir=True): - # We assume keys have a 'namespace.hashedKey' format - parts = key.split('.', 1) - if len(parts) == 1: - namespace = '_' - filename = parts[0] - else: - namespace, filename = parts - dir = os.path.join(self._dir, namespace, filename[:4], filename[:8]) - if mkdir and not os.path.exists(dir): - try: - os.makedirs(dir) - except OSError: - # Handle race condition - if not os.path.exists(dir): - raise - return os.path.join(dir, filename) - - def set(self, key, val, ttl=0): - try: - f = open(self._getFilePath(key), 'wb') - OSSpecific.lockFile(f, 'LOCK_EX') - try: - expiry = int(time.time()) + ttl if ttl else None - data = (expiry, val) - pickle.dump(data, f) - finally: - OSSpecific.lockFile(f, 'LOCK_UN') - f.close() - except (IOError, OSError): - Logger.get('cache.files').exception('Error setting value in cache') - return 0 - return 1 - - def get(self, key): - try: - path = self._getFilePath(key, False) - if not os.path.exists(path): - return None - - f = open(path, 'rb') - OSSpecific.lockFile(f, 'LOCK_SH') - expiry = val = None - try: - expiry, val = pickle.load(f) - finally: - OSSpecific.lockFile(f, 'LOCK_UN') - f.close() - if expiry and time.time() > expiry: - return None - except (IOError, OSError): - Logger.get('cache.files').exception('Error getting cached value') - return None - except (EOFError, pickle.UnpicklingError): - Logger.get('cache.files').exception('Cached information seems corrupted. Overwriting it.') - return None - - return val - - def delete(self, key): - path = self._getFilePath(key, False) - if os.path.exists(path): - silentremove(path) - return 1 - - -class MemcachedCacheClient(CacheClient): - """Memcached-based cache client""" - - @staticmethod - def convert_ttl(ttl): - """Convert a ttl in seconds to a timestamp for use with memcached.""" - return (int(time.time()) + ttl) if ttl else 0 - - def __init__(self, servers): - import memcache - self._client = memcache.Client(servers) - - def set(self, key, val, ttl=0): - return self._client.set(key, val, self.convert_ttl(ttl)) - - def get(self, key): - return self._client.get(key) - - def delete(self, key): - return self._client.delete(key) - - -class GenericCache(object): - """A simple cache interface that supports various backends. - - The backends are accessed through the CacheClient interface. - """ - def __init__(self, namespace): - self._client = None - self._namespace = namespace - - def __repr__(self): - return 'GenericCache(%r)' % self._namespace - - def _connect(self): - """Connect to the CacheClient. - - This method must be called before accessing ``self._client``. - """ - # Maybe we already have a client in this instance - if self._client is not None: - return - # If not, we might have one from another instance - self._client = g.get('generic_cache_client', None) - - if self._client is not None: - return - - # If not, create a new one - backend = config.CACHE_BACKEND - if backend == 'memcached': - self._client = MemcachedCacheClient(config.MEMCACHED_SERVERS) - elif backend == 'redis': - self._client = RedisCacheClient(config.REDIS_CACHE_URL) - elif backend == 'files': - self._client = FileCacheClient(config.CACHE_DIR) - else: - self._client = NullCacheClient() - - g.generic_cache_client = self._client - - def _hashKey(self, key): - if hasattr(self._client, 'hash_key'): - return self._client.hash_key(key) - return hashlib.sha256(key).hexdigest() - - def _makeKey(self, key): - if not isinstance(key, basestring): - # In case we get something not a string (number, list, whatever) - key = repr(key) - # Hashlib doesn't allow unicode so let's ensure it's not! - key = key.encode('utf-8') - return '%s%s.%s' % (getattr(self._client, 'key_prefix', ''), self._namespace, self._hashKey(key)) - - def _processTime(self, ts): - if isinstance(ts, datetime.timedelta): - ts = ts.seconds + (ts.days * 24 * 3600) - return ts - - def set(self, key, val, time=0): - """Set key to val with optional validity time. - - :param key: the key of the cache entry - :param val: any python object that can be pickled - :param time: number of seconds or a datetime.timedelta - """ - self._connect() - time = self._processTime(time) - Logger.get('cache.generic').debug('SET %s %r (%d)', self._namespace, key, time) - self._client.set(self._makeKey(key), _NoneValue.replace(val), time) - - def set_multi(self, mapping, time=0): - self._connect() - time = self._processTime(time) - mapping = dict(((self._makeKey(key), _NoneValue.replace(val)) for key, val in mapping.iteritems())) - self._client.set_multi(mapping, time) - - def get(self, key, default=None): - self._connect() - res = self._client.get(self._makeKey(key)) - Logger.get('cache.generic').debug('GET %s %r (%s)', self._namespace, key, 'HIT' if res is not None else 'MISS') - if res is None: - return default - return _NoneValue.restore(res) - - def get_multi(self, keys, default=None, asdict=True): - self._connect() - real_keys = map(self._makeKey, keys) - data = self._client.get_multi(real_keys) - # Add missing keys - for real_key in real_keys: - if real_key not in data: - data[real_key] = default - # Get data in the same order as our keys - sorted_data = (default if data[rk] is None else _NoneValue.restore(data[rk]) for rk in real_keys) - if asdict: - return dict(izip(keys, sorted_data)) - else: - return list(sorted_data) - - def delete(self, key): - self._connect() - Logger.get('cache.generic').debug('DEL %s %r', self._namespace, key) - self._client.delete(self._makeKey(key)) - - def delete_multi(self, keys): - self._connect() - keys = map(self._makeKey, keys) - self._client.delete_multi(keys) diff --git a/indico/legacy/common/contribPacker.py b/indico/legacy/common/contribPacker.py deleted file mode 100644 index 594a6de2e3c..00000000000 --- a/indico/legacy/common/contribPacker.py +++ /dev/null @@ -1,56 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -import os -import string -import tempfile -import zipfile - -from indico.core.config import config -from indico.legacy.common.utils import utf8rep - - -class ZIPFileHandler: - def __init__(self): - fh, name = tempfile.mkstemp(prefix="Indico", dir=config.TEMP_DIR) - os.fdopen(fh).close() - try: - self._file = zipfile.ZipFile(name, "w", zipfile.ZIP_DEFLATED, allowZip64=True) - except RuntimeError: - self._file = zipfile.ZipFile(name, "w", allowZip64=True) - self._name = name - - def _normalisePath(self, path): - forbiddenChars = string.maketrans(" :()*?<>|\"", "__________") - path = path.translate(forbiddenChars) - return path - - def add(self, name, path): - name = utf8rep(name) - self._file.write(str(path), self._normalisePath(name)) - - def addNewFile(self, name, bytes): - if not self.hasFile(name): - name = utf8rep(name) - self._file.writestr(name, bytes) - - def addDir(self, path): - normalized_path = os.path.join(self._normalisePath(path), "indico_file.dat") - if not self.hasFile(normalized_path): - self.addNewFile(normalized_path, "# Indico File") - - def close(self): - self._file.close() - - def getPath(self): - return self._name - - def hasFile(self, fileName): - for zfile in self._file.infolist(): - if zfile.filename == fileName: - return True - return False diff --git a/indico/legacy/common/output.py b/indico/legacy/common/output.py index 60bd2c3f458..832d374fef3 100644 --- a/indico/legacy/common/output.py +++ b/indico/legacy/common/output.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -7,7 +7,6 @@ # flake8: noqa -import string from collections import defaultdict from datetime import datetime @@ -17,9 +16,7 @@ from indico.modules.attachments.models.attachments import Attachment, AttachmentType from indico.modules.attachments.models.folders import AttachmentFolder from indico.modules.groups import GroupProxy -from indico.modules.groups.legacy import LDAPGroupWrapper from indico.modules.users import User -from indico.modules.users.legacy import AvatarUserWrapper from indico.util.event import uniqueId from indico.web.flask.util import url_for @@ -28,7 +25,7 @@ def get_map_url(item): return item.room.map_url if item.room else None -class outputGenerator(object): +class outputGenerator: def __init__(self, user, XG=None): self.__user = user if XG is not None: @@ -46,9 +43,7 @@ def _getRecordCollection(self, obj): return "INDICOSEARCH.PUBLIC" def _generateACLDatafield(self, eType, memberList, objId, out): - """ - Generates a specific MARCXML 506 field containing the ACL - """ + """Generate a specific MARCXML 506 field containing the ACL.""" if eType: out.openTag("datafield", [["tag", "506"], ["ind1", "1"], ["ind2", " "]]) @@ -75,25 +70,20 @@ def _generateAccessList(self, obj=None, out=None, acl=None, objId=None): obj could be a Conference, Session, Contribution, Material, Resource or SubContribution object. """ - if acl is None: acl = obj.get_access_list() # Populate two lists holding email/group strings instead of - # Avatar/Group objects + # User/Group objects allowed_logins = set() allowed_groups = [] for user_obj in acl: - if isinstance(user_obj, (User, AvatarUserWrapper)): - if isinstance(user_obj, AvatarUserWrapper): - user_obj = user_obj.user + if isinstance(user_obj, User): # user names for all non-local accounts for provider, identifier in user_obj.iter_identifiers(): if provider != 'indico': allowed_logins.add(identifier) - elif isinstance(user_obj, LDAPGroupWrapper): - allowed_groups.append(user_obj.getId()) elif isinstance(user_obj, GroupProxy) and not user_obj.is_local: allowed_groups.append(user_obj.name) @@ -118,7 +108,7 @@ def confToXMLMarc21(self, event, includeSession=1, includeContribution=1, includ out.writeXML(xml) def _generate_category_path(self, event, out): - path = [unicode(c.id) for c in event.category.chain_query.options(load_only('id'))] + path = [str(c.id) for c in event.category.chain_query.options(load_only('id'))] out.openTag("datafield", [["tag", "650"], ["ind1", " "], ["ind2", "7"]]) out.writeTag("subfield", ":".join(path), [["code", "a"]]) out.closeTag("datafield") @@ -146,8 +136,10 @@ def _event_to_xml_marc_21(self, event, includeSession=1, includeContribution=1, sd = event.start_dt ed = event.end_dt - out.writeTag("subfield","%d-%s-%sT%s:%s:00Z" %(sd.year, string.zfill(sd.month,2), string.zfill(sd.day,2), string.zfill(sd.hour,2), string.zfill(sd.minute,2)),[["code","9"]]) - out.writeTag("subfield","%d-%s-%sT%s:%s:00Z" %(ed.year, string.zfill(ed.month,2), string.zfill(ed.day,2), string.zfill(ed.hour,2), string.zfill(ed.minute,2)),[["code","z"]]) + out.writeTag("subfield","%04d-%02d-%02dT%02d:%02d:00Z" % (sd.year, sd.month, sd.day, sd.hour, sd.minute), + [["code","9"]]) + out.writeTag("subfield","%04d-%02d-%02dT%02d:%02d:00Z" % (ed.year, ed.month, ed.day, ed.hour, ed.minute), + [["code","z"]]) out.writeTag("subfield", uniqueId(event), [["code", "g"]]) out.closeTag("datafield") @@ -157,7 +149,8 @@ def _event_to_xml_marc_21(self, event, includeSession=1, includeContribution=1, sd = event.start_dt if sd is not None: out.openTag("datafield",[["tag","518"],["ind1"," "],["ind2"," "]]) - out.writeTag("subfield","%d-%s-%sT%s:%s:00Z" %(sd.year, string.zfill(sd.month,2), string.zfill(sd.day,2), string.zfill(sd.hour,2), string.zfill(sd.minute,2)),[["code","d"]]) + out.writeTag("subfield","%04d-%02d-%02dT%02d:%02d:00Z" % (sd.year, sd.month, sd.day, sd.hour, sd.minute), + [["code","d"]]) out.closeTag("datafield") out.openTag("datafield",[["tag","520"],["ind1"," "],["ind2"," "]]) @@ -301,11 +294,11 @@ def _contrib_to_marc_xml_21(self, contrib, include_material=1, out=None): self.noteToXMLMarc21(contrib.note, out=out) out.openTag("datafield",[["tag","962"],["ind1"," "],["ind2"," "]]) - out.writeTag("subfield", 'INDICO.{}'.format(uniqueId(contrib.event)), [["code", "b"]]) + out.writeTag("subfield", f'INDICO.{uniqueId(contrib.event)}', [["code", "b"]]) out.closeTag("datafield") out.openTag("datafield",[["tag","970"],["ind1"," "],["ind2"," "]]) - out.writeTag("subfield", 'INDICO.{}'.format(uniqueId(contrib)), [["code", "a"]]) + out.writeTag("subfield", f'INDICO.{uniqueId(contrib)}', [["code", "a"]]) out.closeTag("datafield") out.openTag("datafield",[["tag","980"],["ind1"," "],["ind2"," "]]) @@ -353,10 +346,10 @@ def _generate_contrib_people(self, contrib, out, subcontrib=None): users[user].append('Author') for user in sList: users[user].append('Speaker') - for user, roles in users.iteritems(): + for user, roles in users.items(): tag = '100' if user in contrib.primary_authors else '700' out.openTag('datafield', [['tag', tag], ['ind1', ' '], ['ind2', ' ']]) - out.writeTag('subfield', u'{} {}'.format(user.last_name, user.first_name), [['code', 'a']]) + out.writeTag('subfield', f'{user.last_name} {user.first_name}', [['code', 'a']]) for role in roles: out.writeTag('subfield', role, [['code', 'e']]) out.writeTag('subfield', user.affiliation, [['code', 'u']]) @@ -376,7 +369,7 @@ def _subcontrib_to_marc_xml_21(self, subcontrib, includeMaterial=1, out=None): out.writeTag("leader", "00000nmm 2200000uu 4500") out.openTag("datafield",[["tag","035"],["ind1"," "],["ind2"," "]]) - out.writeTag("subfield", 'INDICO.{}'.format(uniqueId(subcontrib)), [["code", "a"]]) + out.writeTag("subfield", f'INDICO.{uniqueId(subcontrib)}', [["code", "a"]]) out.closeTag("datafield") # out.openTag("datafield",[["tag","035"],["ind1"," "],["ind2"," "]]) @@ -423,11 +416,11 @@ def _subcontrib_to_marc_xml_21(self, subcontrib, includeMaterial=1, out=None): self.noteToXMLMarc21(subcontrib.note, out=out) out.openTag("datafield",[["tag","962"],["ind1"," "],["ind2"," "]]) - out.writeTag("subfield", 'INDICO.{}'.format(uniqueId(subcontrib.event)), [["code", "b"]]) + out.writeTag("subfield", f'INDICO.{uniqueId(subcontrib.event)}', [["code", "b"]]) out.closeTag("datafield") out.openTag("datafield",[["tag","970"],["ind1"," "],["ind2"," "]]) - confcont = 'INDICO.{}'.format(uniqueId(subcontrib)) + confcont = f'INDICO.{uniqueId(subcontrib)}' out.writeTag("subfield",confcont,[["code","a"]]) out.closeTag("datafield") @@ -443,9 +436,13 @@ def _subcontrib_to_marc_xml_21(self, subcontrib, includeMaterial=1, out=None): def materialToXMLMarc21(self, obj, out=None): if not out: out = self._XMLGen - for attachment in (Attachment.find(~AttachmentFolder.is_deleted, AttachmentFolder.object == obj, - is_deleted=False, _join=AttachmentFolder) - .options(joinedload(Attachment.legacy_mapping))): + query = (Attachment.query + .filter(~AttachmentFolder.is_deleted, + AttachmentFolder.object == obj, + ~Attachment.is_deleted) + .join(AttachmentFolder) + .options(joinedload(Attachment.legacy_mapping))) + for attachment in query: if attachment.can_access(self.__user): self.resourceToXMLMarc21(attachment, out) self._generateAccessList(acl=self._attachment_access_list(attachment), out=out, @@ -461,11 +458,11 @@ def resourceToXMLMarc21(self, res, out=None): def _attachment_unique_id(self, attachment, add_prefix=True): if attachment.legacy_mapping: - unique_id = 'm{}.{}'.format(attachment.legacy_mapping.material_id, attachment.legacy_mapping.resource_id) + unique_id = f'm{attachment.legacy_mapping.material_id}.{attachment.legacy_mapping.resource_id}' else: - unique_id = 'a{}'.format(attachment.id) - unique_id = '{}{}'.format(uniqueId(attachment.folder.object), unique_id) - return 'INDICO.{}'.format(unique_id) if add_prefix else unique_id + unique_id = f'a{attachment.id}' + unique_id = f'{uniqueId(attachment.folder.object)}{unique_id}' + return f'INDICO.{unique_id}' if add_prefix else unique_id def _attachment_access_list(self, attachment): linked_object = attachment.folder.object @@ -511,7 +508,7 @@ def noteToXMLMarc21(self, note, out=None): out = self._XMLGen out.openTag('datafield', [['tag', '856'], ['ind1', '4'], ['ind2', ' ']]) out.writeTag('subfield', url_for('event_notes.view', note, _external=True), [['code', 'u']]) - out.writeTag('subfield', u'{} - Minutes'.format(note.object.title), [['code', 'y']]) - out.writeTag('subfield', 'INDICO.{}'.format(uniqueId(note)), [['code', '3']]) + out.writeTag('subfield', f'{note.object.title} - Minutes', [['code', 'y']]) + out.writeTag('subfield', f'INDICO.{uniqueId(note)}', [['code', '3']]) out.writeTag('subfield', 'resource', [['code', 'x']]) out.closeTag('datafield') diff --git a/indico/legacy/common/utils.py b/indico/legacy/common/utils.py index ebb7e9eac46..23c5f76ac5c 100644 --- a/indico/legacy/common/utils.py +++ b/indico/legacy/common/utils.py @@ -1,87 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -import os - - -# fcntl is only available for POSIX systems -if os.name == 'posix': - import fcntl - - -def utf8rep(text): - # \x -> _x keeps windows systems satisfied - return text.decode('utf-8').encode('unicode_escape').replace('\\x', '_x') - - def isStringHTML(s): - if not isinstance(s, basestring): + if not isinstance(s, str): return False s = s.lower() return any(tag in s for tag in ('

', '

')) - - -def encodeUnicode(text, sourceEncoding="utf-8"): - try: - tmp = str(text).decode(sourceEncoding) - except UnicodeError: - try: - tmp = str(text).decode('iso-8859-1') - except UnicodeError: - return "" - return tmp.encode('utf-8') - - -def unicodeSlice(s, start, end, encoding='utf-8'): - """Returns a slice of the string s, based on its encoding.""" - return s.decode(encoding, 'replace')[start:end] - - -class OSSpecific(object): - """ - Namespace for OS Specific operations: - - file locking - """ - - @classmethod - def _lockFilePosix(cls, f, lockType): - """ - Locks file f with lock type lockType - """ - fcntl.flock(f, lockType) - - @classmethod - def _lockFileOthers(cls, f, lockType): - """ - Win32/others file locking could be implemented here - """ - pass - - @classmethod - def lockFile(cls, f, lockType): - """ - API method - locks a file - f - file handler - lockType - string: LOCK_EX | LOCK_UN | LOCK_SH - """ - cls._lockFile(f, cls._lockTranslationTable[lockType]) - - # Check OS and choose correct locking method - if os.name == 'posix': - _lockFile = _lockFilePosix - _lockTranslationTable = { - 'LOCK_EX': fcntl.LOCK_EX, - 'LOCK_UN': fcntl.LOCK_UN, - 'LOCK_SH': fcntl.LOCK_SH - } - else: - _lockFile = _lockFileOthers - _lockTranslationTable = { - 'LOCK_EX': None, - 'LOCK_UN': None, - 'LOCK_SH': None - } diff --git a/indico/legacy/common/xmlGen.py b/indico/legacy/common/xmlGen.py index 226a6fb5cfd..1b416a1a604 100644 --- a/indico/legacy/common/xmlGen.py +++ b/indico/legacy/common/xmlGen.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -9,32 +9,24 @@ from xml.sax import saxutils -from indico.legacy.common.utils import encodeUnicode -from indico.util.string import encode_if_unicode - class XMLGen: def __init__(self, init=True): - self.setSourceEncoding( "utf-8" ) if init: self.initXml() else: self.xml=[] self.indent = 0 - def setSourceEncoding( self, newEnc = "utf-8" ): - self._sourceEncoding = newEnc - def initXml(self): self.xml=["""\n"""] def getXml(self): return "".join(self.xml) - def escapeString(self,text): - tmp = encodeUnicode(text, self._sourceEncoding) - return saxutils.escape( tmp ) + def escapeString(self, text): + return saxutils.escape(text) def openTag(self, name, listAttrib=[], single=False): #open an XML tag @@ -85,4 +77,4 @@ def cleanText(self, text): cm.append(" ") else: cm.append(chr(c)) - return str(encode_if_unicode(text)).translate("".join(cm)) + return str(text).translate("".join(cm)) diff --git a/indico/legacy/fossils/user.py b/indico/legacy/fossils/user.py deleted file mode 100644 index 6473d488829..00000000000 --- a/indico/legacy/fossils/user.py +++ /dev/null @@ -1,72 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from indico.util.fossilize import IFossil - - -class IGroupFossil(IFossil): - - def getId(self): - """ Group id """ - - def getName(self): - """ Group name """ - - def getEmail(self): - """ Group email """ - - def getProvider(self): - pass - getProvider.produce = lambda x: getattr(x, 'provider', None) - - def getIdentifier(self): - pass - getIdentifier.produce = lambda x: 'Group:{}:{}'.format(getattr(x, 'provider', ''), x.id) - - -class IAvatarMinimalFossil(IFossil): - - def getId(self): - """ Avatar id""" - - def getIdentifier(self): - pass - getIdentifier.produce = lambda x: 'User:{}'.format(x.id) - - def getStraightFullName(self): - """ Avatar full name, the one usually displayed """ - getStraightFullName.name = "name" - getStraightFullName.produce = lambda x: x.getStraightFullName(upper=False) - - -class IAvatarFossil(IAvatarMinimalFossil): - - def getEmail(self): - """ Avatar email """ - - def getFirstName(self): - """ Avatar first name """ - - def getFamilyName(self): - """ Avatar family name """ - - def getTitle(self): - """ Avatar name title (Mr, Mrs..) """ - - def getTelephone(self): - """ Avatar telephone """ - getTelephone.name = "phone" - - def getOrganisation(self): - """ Avatar organisation / affiliation """ - getOrganisation.name = "affiliation" - - def getFax(self): - """ Avatar fax """ - - def getAddress(self): - """ Avatar address """ diff --git a/indico/legacy/pdfinterface/base.py b/indico/legacy/pdfinterface/base.py index 15cc8ca3fa2..9cb1271e625 100644 --- a/indico/legacy/pdfinterface/base.py +++ b/indico/legacy/pdfinterface/base.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -11,6 +11,7 @@ import math import os import xml.sax.saxutils as saxutils +from io import BytesIO import pkg_resources from PIL import Image as PILImage @@ -28,7 +29,7 @@ from indico.legacy.common.utils import isStringHTML from indico.util.i18n import _ -from indico.util.string import sanitize_for_platypus, to_unicode +from indico.util.string import sanitize_for_platypus ratio = math.sqrt(math.sqrt(2.0)) @@ -145,10 +146,12 @@ def int_to_roman(value): class Paragraph(platypus.Paragraph): """ - add a part attribute for drawing the name of the current part on the laterPages function + Add a part attribute for drawing the name of the current part + on the laterPages function. """ + def __init__(self, text, style, part="", bulletText=None, frags=None, caseSensitive=1): - platypus.Paragraph.__init__(self, to_unicode(text), style, bulletText, frags, caseSensitive) + platypus.Paragraph.__init__(self, str(text), style, bulletText, frags, caseSensitive) self._part = part def setPart(self, part): @@ -158,11 +161,12 @@ def getPart(self): return self._part class SimpleParagraph(platypus.Flowable): - """ Simple and fast paragraph. + """Simple and fast paragraph. WARNING! This paragraph cannot break the line and doesn't have almost any formatting methods. It's used only to increase PDF performance in places where normal paragraph is not needed. """ + def __init__(self, text, fontSize = 10, indent = 0, spaceAfter = 2): platypus.Flowable.__init__(self) self.text = text @@ -180,18 +184,19 @@ def draw(self): self.canv.drawString(self.indent, self.spaceAfter, self.text) class TableOfContentsEntry(Paragraph): + """Class used to create table of contents entry with its number. + + Much faster than table of table of contents from platypus lib """ - Class used to create table of contents entry with its number. - Much faster than table of table of contents from platypus lib - """ + def __init__(self, text, pageNumber, style, part="", bulletText=None, frags=None, caseSensitive=1): - Paragraph.__init__(self, to_unicode(text), style, part, bulletText, frags, caseSensitive) + Paragraph.__init__(self, str(text), style, part, bulletText, frags, caseSensitive) self._part = part self._pageNumber = pageNumber def _drawDots(self): """ - Draws row of dots from the end of the abstract title to the page number. + Draw row of dots from the end of the abstract title to the page number. """ try: freeSpace = int(self.blPara.lines[-1][0]) @@ -259,20 +264,6 @@ def getPart(self): return self._part -class FileDummy: - def __init__(self): - self._data = "" - self.name = "fileDummy" - - def write(self, data): - self._data += data - - def getData(self): - return self._data - - def close(self): - pass - class CanvasA0(Canvas): def __init__(self,filename, pagesize=None, @@ -363,7 +354,7 @@ def __init__(self, doc=None, story=None, pagesize = 'A4', printLandscape=False, #create a new document #As the constructor of SimpleDocTemplate can take only a filename or a file object, #to keep the PDF data not in a file, we use a dummy file object which save the data in a string - self._fileDummy = FileDummy() + self._fileDummy = BytesIO() if printLandscape: self._doc = SimpleDocTemplate(self._fileDummy, pagesize=landscape(PDFSizes().PDFpagesizes[pagesize])) else: @@ -390,28 +381,26 @@ def __init__(self, doc=None, story=None, pagesize = 'A4', printLandscape=False, setTTFonts() def firstPage(self, c, doc): - """set the first page of the document - This function is call by doc.build method for the first page + """Set the first page of the document. + + This function is called by doc.build method for the first page. """ - pass def laterPages(self, c, doc): - """set the layout of the page after the first - This function is call by doc.build method one each page after the first + """Set the layout of the page after the first. + + This function is called by doc.build method one each page after the first. """ - pass def getBody(self, story=None): - """add the content to the story - """ - pass + """Add the content to the story.""" def getPDFBin(self): #build the pdf in the fileDummy self.getBody() self._doc.build(self._story, onFirstPage=self.firstPage, onLaterPages=self.laterPages) #return the data from the fileDummy - return self._fileDummy.getData() + return self._fileDummy.getvalue() def _drawWrappedString(self, c, text, font='Times-Bold', size=30, color=(0, 0, 0), align="center", width=None, height=None, measurement=cm, lineSpacing=1, maximumWidth=None): @@ -462,8 +451,7 @@ def _drawLogo(self, c, drawTitle = True): startHeight = self._PAGE_HEIGHT if drawTitle: - startHeight = self._drawWrappedString(c, escape(self.event.title.encode('utf-8')), - height=self._PAGE_HEIGHT - inch) + startHeight = self._drawWrappedString(c, escape(self.event.title), height=self._PAGE_HEIGHT - inch) # lower edge of the image startHeight = startHeight - inch / 2 - height @@ -471,16 +459,14 @@ def _drawLogo(self, c, drawTitle = True): # draw horizontally centered, with recalculated width and height c.drawImage(imagePath, self._PAGE_WIDTH/2.0 - width/2, startHeight, width, height, mask="auto") return startHeight - except IOError: + except OSError: if drawTitle: - self._drawWrappedString(c, escape(self.event.title.encode('utf-8')), - height=self._PAGE_HEIGHT - inch) + self._drawWrappedString(c, escape(self.event.title), height=self._PAGE_HEIGHT - inch) return 0 def _doNothing(canvas, doc): - "Dummy callback for onPage" - pass + "Dummy callback for onPage." class DocTemplateWithTOC(SimpleDocTemplate): @@ -532,7 +518,7 @@ def _prepareTOC(self): self._tocStory.append(Spacer(inch, 2*cm)) for entry in self._toc: indent = ((entry[0] - 1) * 50) - toc_entry = TableOfContentsEntry('{}'.format(indent, entry[1]), + toc_entry = TableOfContentsEntry(f'{entry[1]}', str(entry[2]), entryStyle) self._tocStory.append(toc_entry) @@ -552,7 +538,7 @@ def multiBuild(self, story, filename=None, canvasMaker=Canvas, maxPasses=10, onF SimpleDocTemplate.multiBuild(self, story, maxPasses, canvasmaker=canvasMaker) self._prepareTOC() contentFile = self.filename - self.filename = FileDummy() + self.filename = BytesIO() self.pageTemplates = [] self.addPageTemplates([PageTemplate(id='First',frames=frameT, onPage=onFirstPage,pagesize=self.pagesize)]) if onFirstPage is _doNothing and hasattr(self,'onFirstPage'): @@ -564,33 +550,26 @@ def multiBuild(self, story, filename=None, canvasMaker=Canvas, maxPasses=10, onF self.mergePDFs(self.filename, contentFile) def mergePDFs(self, pdf1, pdf2): - from pyPdf import PdfFileWriter, PdfFileReader - import cStringIO - outputStream = cStringIO.StringIO() - pdf1Stream = cStringIO.StringIO() - pdf2Stream = cStringIO.StringIO() - pdf1Stream.write(pdf1.getData()) - pdf2Stream.write(pdf2.getData()) + from PyPDF2 import PdfFileReader, PdfFileWriter output = PdfFileWriter() - background_pages = PdfFileReader(pdf1Stream) - foreground_pages = PdfFileReader(pdf2Stream) + background_pages = PdfFileReader(pdf1) + foreground_pages = PdfFileReader(pdf2) for page in background_pages.pages: output.addPage(page) for page in foreground_pages.pages: output.addPage(page) - output.write(outputStream) - pdf2._data = outputStream.getvalue() - outputStream.close() + outputbuf = BytesIO() + output.write(outputbuf) + pdf2.seek(0) + pdf2.truncate() + pdf2.write(outputbuf.getvalue()) def getCurrentPart(self): return self._part class PDFWithTOC(PDFBase): - """ - create a PDF with a Table of Contents - - """ + """Create a PDF with a Table of Contents.""" def __init__(self, story=None, pagesize='A4', fontsize='normal', firstPageNumber=1, include_toc=True): self._fontsize = fontsize @@ -601,7 +580,7 @@ def __init__(self, story=None, pagesize='A4', fontsize='normal', firstPageNumber self._story.append(Spacer(inch, 0*cm)) self._indexedFlowable = {} - self._fileDummy = FileDummy() + self._fileDummy = BytesIO() self._doc = DocTemplateWithTOC(self._indexedFlowable, self._fileDummy, firstPageNumber=firstPageNumber, pagesize=PDFSizes().PDFpagesizes[pagesize], @@ -613,9 +592,9 @@ def __init__(self, story=None, pagesize='A4', fontsize='normal', firstPageNumber setTTFonts() def _processTOCPage(self): - """ Generates page with table of contents. + """Generates page with table of contents. - Not used, because table of contents is generated automatically inside DocTemplateWithTOC class + Not used, because table of contents is generated automatically inside DocTemplateWithTOC class. """ style1 = ParagraphStyle({}) style1.fontName = "Times-Bold" @@ -630,15 +609,14 @@ def _processTOCPage(self): self._story.append(PageBreak()) def getBody(self, story=None): - """add the content to the story + """Add the content to the story. When you want to put a paragraph p in the toc, add it to the self._indexedFlowable as this: self._indexedFlowable[p] = {"text":"my title", "level":1} """ if not story: story = self._story - pass def getPDFBin(self): self.getBody() self._doc.multiBuild(self._story, onFirstPage=self.firstPage, onLaterPages=self.laterPages) - return self._fileDummy.getData() + return self._fileDummy.getvalue() diff --git a/indico/legacy/pdfinterface/conference.py b/indico/legacy/pdfinterface/conference.py index 471f9ed9410..9e806cc19e1 100644 --- a/indico/legacy/pdfinterface/conference.py +++ b/indico/legacy/pdfinterface/conference.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -23,7 +23,6 @@ from speaklater import is_lazy_string from indico.core.db import db -from indico.legacy.common import utils from indico.legacy.pdfinterface.base import PageBreak, Paragraph, PDFBase, PDFWithTOC, Spacer, escape, modifiedFontSize from indico.modules.events.layout.util import get_menu_entry_by_name from indico.modules.events.registration.models.items import PersonalDataType @@ -33,7 +32,7 @@ from indico.util.date_time import format_date, format_datetime, format_human_timedelta, format_time, now_utc from indico.util.i18n import _, ngettext from indico.util.string import (format_full_name, html_color_to_rgb, natural_sort_key, render_markdown, - sanitize_for_platypus, strip_tags, to_unicode, truncate) + sanitize_for_platypus, strip_tags, truncate) # Change reportlab default pdf font Helvetica to indico ttf font, @@ -51,7 +50,7 @@ def _get_sans_style_sheet(): } styles = getSampleStyleSheet() - for name, style in styles.byName.iteritems(): + for name, style in styles.byName.items(): if hasattr(style, 'fontName'): style.fontName = _font_map.get(style.fontName, style.fontName) if hasattr(style, 'bulletFontName'): @@ -64,7 +63,7 @@ class ProgrammeToPDF(PDFBase): def __init__(self, event, tz=None): self.event = event self._tz = self.event.tzinfo - self._title = get_menu_entry_by_name('program', event).localized_title.encode('utf-8') + self._title = get_menu_entry_by_name('program', event).localized_title PDFBase.__init__(self, title='program.pdf') def firstPage(self, c, doc): @@ -73,7 +72,7 @@ def firstPage(self, c, doc): height = self._drawLogo(c) if not height: - height = self._drawWrappedString(c, escape(strip_tags(self.event.title).encode('utf-8')), + height = self._drawWrappedString(c, escape(strip_tags(self.event.title)), height=self._PAGE_HEIGHT - 2*inch) c.setFont('Times-Bold', 15) @@ -84,18 +83,18 @@ def firstPage(self, c, doc): self.event.start_dt_local.strftime("%A %d %B %Y"), self.event.end_dt_local.strftime("%A %d %B %Y"))) if self.event.venue_name: height-=1*cm - c.drawCentredString(self._PAGE_WIDTH / 2.0, height, escape(self.event.venue_name.encode('utf-8'))) + c.drawCentredString(self._PAGE_WIDTH / 2.0, height, escape(self.event.venue_name)) c.setFont('Times-Bold', 30) height-=6*cm c.drawCentredString(self._PAGE_WIDTH/2.0, height, self._title) - self._drawWrappedString(c, "%s / %s" % (strip_tags(self.event.title).encode('utf-8'), self._title), + self._drawWrappedString(c, f"{strip_tags(self.event.title)} / {self._title}", width=inch, height=0.75*inch, font='Times-Roman', size=9, color=(0.5,0.5,0.5), align="left", maximumWidth=self._PAGE_WIDTH-3.5*inch, measurement=inch, lineSpacing=0.15) c.drawRightString(self._PAGE_WIDTH - inch, 0.75 * inch, now_utc().strftime("%A %d %B %Y")) c.restoreState() def laterPages(self, c, doc): c.saveState() - self._drawWrappedString(c, "%s / %s" % (escape(strip_tags(self.event.title).encode('utf-8')), self._title), + self._drawWrappedString(c, f"{escape(strip_tags(self.event.title))} / {self._title}", width=inch, height=self._PAGE_HEIGHT-0.75*inch, font='Times-Roman', size=9, color=(0.5, 0.5, 0.5), align="left", maximumWidth=self._PAGE_WIDTH - 3.5*inch, measurement=inch, lineSpacing=0.15) @@ -116,25 +115,25 @@ def getBody(self, story=None): for i, part in enumerate(re.split(r'\n+', event_program)): if i > 0 and re.match(r'<(p|ul|ol)\b[^>]*>', part): # extra spacing before a block-level element - parts.append(u'
') + parts.append('
') parts.append(part) - story.append(Paragraph(u'\n
\n'.join(parts).encode('utf-8'), style)) + story.append(Paragraph('\n
\n'.join(parts), style)) story.append(Spacer(1, 0.4*inch)) items = self.event.get_sorted_tracks() for item in items: - bogustext = item.title.encode('utf-8') + bogustext = item.title p = Paragraph(escape(bogustext), styles["Heading1"]) self._story.append(p) - bogustext = item.description.encode('utf-8') + bogustext = item.description p = Paragraph(escape(bogustext), style) story.append(p) if isinstance(item, TrackGroup) and item.tracks: for track in item.tracks: - bogustext = track.title.encode('utf-8') + bogustext = track.title p = Paragraph(escape(bogustext), styles["Heading2"]) self._story.append(p) - bogustext = track.description.encode('utf-8') + bogustext = track.description p = Paragraph(escape(bogustext), style) story.append(p) story.append(Spacer(1, 0.4*inch)) @@ -272,7 +271,7 @@ def firstPage(self, c, doc): if self._ttPDFFormat.showLogo(): self._drawLogo(c, False) - height = self._drawWrappedString(c, self.event.title.encode('utf-8')) + height = self._drawWrappedString(c, self.event.title) c.setFont('Times-Bold', modifiedFontSize(15, self._fontsize)) height -= 2 * cm c.drawCentredString(self._PAGE_WIDTH / 2.0, height, @@ -284,7 +283,7 @@ def firstPage(self, c, doc): c.setFont('Times-Bold', modifiedFontSize(30, self._fontsize)) height -= 1 * cm c.drawCentredString(self._PAGE_WIDTH / 2.0, height, self._title) - self._drawWrappedString(c, "{} / {}".format(self.event.title.encode('utf-8'), self._title), + self._drawWrappedString(c, f"{self.event.title} / {self._title}", width=inch, height=0.75 * inch, font='Times-Roman', size=modifiedFontSize(9, self._fontsize), color=(0.5, 0.5, 0.5), align="left", maximumWidth=self._PAGE_WIDTH - 3.5 * inch, @@ -297,7 +296,7 @@ def laterPages(self, c, doc): maxi = self._PAGE_WIDTH - 2 * cm if doc.getCurrentPart().strip(): maxi = self._PAGE_WIDTH - 6 * cm - self._drawWrappedString(c, "{} / {}".format(self.event.title.encode('utf-8'), self._title), + self._drawWrappedString(c, f"{self.event.title} / {self._title}", width=1 * cm, height=self._PAGE_HEIGHT - 1 * cm, font='Times-Roman', size=modifiedFontSize(9, self._fontsize), color=(0.5, 0.5, 0.5), align="left", @@ -340,14 +339,14 @@ def _defineStyles(self): self._styles["subContrib"] = subContStyle def _getSessionColor(self, block): - session_color = '#{}'.format(block.session.colors.background) + session_color = f'#{block.session.colors.background}' return html_color_to_rgb(session_color) def _get_speaker_name(self, speaker): speaker_name = speaker.get_full_name(last_name_first=True, show_title=self._ttPDFFormat.showSpeakerTitle(), abbrev_first_name=False) if self._showSpeakerAffiliation and speaker.affiliation: - speaker_name += u' ({})'.format(speaker.affiliation) + speaker_name += f' ({speaker.affiliation})' return speaker_name def _processContribution(self, contrib, l): @@ -356,28 +355,28 @@ def _processContribution(self, contrib, l): lt = [] date = format_time(contrib.start_dt, timezone=self._tz) - caption = u'[{}] {}'.format(contrib.friendly_id, escape(contrib.title)) + caption = f'[{contrib.friendly_id}] {escape(contrib.title)}' if not self._ttPDFFormat.showContribId(): caption = escape(contrib.title) elif self._ttPDFFormat.showLengthContribs(): - caption = u"{} ({})".format(caption, format_human_timedelta(contrib.timetable_entry.duration)) + caption = f"{caption} ({format_human_timedelta(contrib.timetable_entry.duration)})" elif self._ttPDFFormat.showContribAbstract(): - caption = u'{}'.format(caption) + caption = f'{caption}' color_cell = "" - caption = u'{}'.format(unicode(modifiedFontSize(10, self._fontsize)), caption) - lt.append([self._fontify(caption.encode('utf-8'), 10)]) + caption = f'{caption}' + lt.append([self._fontify(caption, 10)]) if self._useColors(): color_cell = " " if self._ttPDFFormat.showContribAbstract(): speaker_list = [self._get_speaker_name(spk) for spk in contrib.speakers] if speaker_list: - speaker_title = u' {}: '.format(ngettext(u'Presenter', u'Presenters', len(speaker_list))) - speaker_content = speaker_title + u', '.join(speaker_list) - speaker_content = u'{}'.format(speaker_content) + speaker_title = ' {}: '.format(ngettext('Presenter', 'Presenters', len(speaker_list))) + speaker_content = speaker_title + ', '.join(speaker_list) + speaker_content = f'{speaker_content}' lt.append([self._fontify(speaker_content, 9)]) - caption = escape(unicode(contrib.description)) + caption = escape(str(contrib.description)) lt.append([self._fontify(caption, 9)]) caption_and_speakers = Table(lt, colWidths=None, style=self._tsSpk) if color_cell: @@ -401,17 +400,17 @@ def _processContribution(self, contrib, l): return lt = [] - caption = '- [{}] {}'.format(subc.friendly_id, escape(subc.title.encode('utf-8'))) + caption = f'- [{subc.friendly_id}] {escape(subc.title)}' if not self._ttPDFFormat.showContribId(): - caption = '- {}' .format(escape(subc.title.encode('utf-8'))) + caption = '- {}' .format(escape(subc.title)) elif self._ttPDFFormat.showLengthContribs(): - caption = '{} ({})'.format(caption, format_human_timedelta(subc.timetable_entry.duration)) + caption = f'{caption} ({format_human_timedelta(subc.timetable_entry.duration)})' - caption = '{}'.format(str(modifiedFontSize(10, self._fontsize)), caption) + caption = f'{caption}' lt.append([Paragraph(caption, self._styles["subContrib"])]) if self._ttPDFFormat.showContribAbstract(): caption = '{}'.format(str(modifiedFontSize(9, self._fontsize)), - escape(unicode(subc.description))) + escape(str(subc.description))) lt.append([Paragraph(caption, self._styles["subContrib"])]) speaker_list = [[Paragraph(escape(self._get_speaker_name(spk)), self._styles["table_body"])] @@ -438,22 +437,22 @@ def _processPosterContribution(self, contrib, l): return lt = [] - caption_text = u"[{}] {}".format(contrib.friendly_id, escape(contrib.title)) + caption_text = f"[{contrib.friendly_id}] {escape(contrib.title)}" if not self._ttPDFFormat.showContribId(): caption_text = escape(contrib.title) if self._ttPDFFormat.showLengthContribs(): - caption_text = u"{} ({})".format(caption_text, format_human_timedelta(contrib.duration)) - caption_text = u'{}'.format(caption_text) - lt.append([self._fontify(caption_text.encode('utf-8'), 10)]) + caption_text = f"{caption_text} ({format_human_timedelta(contrib.duration)})" + caption_text = f'{caption_text}' + lt.append([self._fontify(caption_text, 10)]) board_number = contrib.board_number if self._ttPDFFormat.showContribAbstract() and self._ttPDFFormat.showContribPosterAbstract(): speaker_list = [self._get_speaker_name(spk) for spk in contrib.speakers] if speaker_list: - speaker_word = u'{}: '.format(ngettext(u'Presenter', u'Presenters', len(speaker_list))) + speaker_word = '{}: '.format(ngettext('Presenter', 'Presenters', len(speaker_list))) speaker_text = speaker_word + ', '.join(speaker_list) - speaker_text = u'{}'.format(speaker_text) + speaker_text = f'{speaker_text}' lt.append([self._fontify(speaker_text, 10)]) - caption_text = escape(unicode(contrib.description)) + caption_text = escape(str(contrib.description)) lt.append([self._fontify(caption_text, 9)]) caption_and_speakers = Table(lt, colWidths=None, style=self._tsSpk) if self._useColors(): @@ -476,15 +475,15 @@ def _processPosterContribution(self, contrib, l): return lt = [] - caption_text = "- [{}] {}".format(subc.friendly_id, escape(subc.title.encode('utf-8'))) + caption_text = f"- [{subc.friendly_id}] {escape(subc.title)}" if not self._ttPDFFormat.showContribId(): - caption_text = "- {}".format(subc.friendly_id) + caption_text = f"- {subc.friendly_id}" if self._ttPDFFormat.showLengthContribs(): - caption_text = "{} ({})".format(caption_text, escape(format_human_timedelta(subc.duration))) + caption_text = f"{caption_text} ({escape(format_human_timedelta(subc.duration))})" lt.append([Paragraph(caption_text, self._styles["subContrib"])]) if self._ttPDFFormat.showContribAbstract(): - caption_text = u'{}'.format(str(modifiedFontSize(9, self._fontsize)), - escape(unicode(subc.description))) + caption_text = '{}'.format(str(modifiedFontSize(9, self._fontsize)), + escape(str(subc.description))) lt.append([Paragraph(caption_text, self._styles["subContrib"])]) speaker_list = [[Paragraph(escape(self._get_speaker_name(spk)), self._styles["table_body"])] for spk in subc.speakers] @@ -541,36 +540,37 @@ def _processDayEntries(self, day, story): if not sess_block.can_access(self._user): continue - room = u'' + room = '' if sess_block.room_name: - room = u' - {}'.format(escape(sess_block.room_name)) + room = f' - {escape(sess_block.room_name)}' session_caption = sess_block.full_title conv = [] for c in sess_block.person_links: if self._showSpeakerAffiliation and c.affiliation: - conv.append(u"{} ({})".format(escape(c.get_full_name(last_name_first=True, - last_name_upper=False, - abbrev_first_name=False)), + conv.append("{} ({})".format(escape(c.get_full_name(last_name_first=True, + last_name_upper=False, + abbrev_first_name=False)), escape(c.affiliation))) else: conv.append(escape(c.full_name)) - conv = u'; '.join(conv) + conv = '; '.join(conv) if conv: - conv = u'-{}: {}'.format(_(u"Conveners"), conv) + conv = '-{}: {}'.format(_("Conveners"), conv) res.append(Paragraph('', self._styles["session_title"])) if self._ttPDFFormat.showDateCloseToSessions(): - start_dt = to_unicode(format_datetime(sess_block.timetable_entry.start_dt, timezone=self._tz)) + start_dt = format_datetime(sess_block.timetable_entry.start_dt, timezone=self._tz) else: - start_dt = to_unicode(format_time(sess_block.timetable_entry.start_dt, timezone=self._tz)) + start_dt = format_time(sess_block.timetable_entry.start_dt, timezone=self._tz) - sess_caption = u'{}'.format(escape(session_caption)) - text = u'{}{} ({}-{})'.format( + sess_caption = f'{escape(session_caption)}' + text = '{}{} ({}-{})'.format( sess_caption, room, start_dt, - to_unicode(format_time(sess_block.timetable_entry.end_dt, timezone=self._tz))) + format_time(sess_block.timetable_entry.end_dt, timezone=self._tz) + ) p1 = Paragraph(text, self._styles["session_title"]) if self._useColors(): @@ -580,14 +580,14 @@ def _processDayEntries(self, day, story): res.append(p1) if self._ttPDFFormat.showTitleSessionTOC(): - self._indexedFlowable[p1] = {'text': escape(sess_block.session.title.encode('utf-8')), 'level': 2} + self._indexedFlowable[p1] = {'text': escape(sess_block.session.title), 'level': 2} # add session description if self._showSessionDescription and sess_block.session.description: - text = u'{}'.format(escape(unicode(sess_block.session.description))) + text = f'{escape(str(sess_block.session.description))}' res.append(Paragraph(text, self._styles["session_description"])) - p2 = Paragraph(conv.encode('utf-8'), self._styles["conveners"]) + p2 = Paragraph(conv, self._styles["conveners"]) res.append(p2) l = [] ts = deepcopy(originalts) @@ -622,14 +622,14 @@ def _processDayEntries(self, day, story): self._processContribution(obj, l) elif s_entry.type == TimetableEntryType.BREAK: lt = [] - date = self._fontify('{}'.format(format_time(s_entry.start_dt, timezone=self._tz))) + date = self._fontify(format_time(s_entry.start_dt, timezone=self._tz)) if self._ttPDFFormat.showLengthContribs(): - caption = u'{} ({})'.format(obj.title, format_human_timedelta(s_entry.duration)) + caption = f'{obj.title} ({format_human_timedelta(s_entry.duration)})' else: caption = obj.title - lt.append([self._fontify(caption.encode('utf-8'), 10)]) + lt.append([self._fontify(caption, 10)]) caption = Table(lt, colWidths=None, style=self._tsSpk) if self._ttPDFFormat.showContribAbstract(): @@ -668,27 +668,27 @@ def _processDayEntries(self, day, story): if not contrib.can_access(self._user): continue - room = u'' + room = '' if contrib.room_name: - room = u' - {}'.format(escape(contrib.room_name)) + room = f' - {escape(contrib.room_name)}' - speakers = u'; '.join([self._get_speaker_name(spk) for spk in contrib.speakers]) + speakers = '; '.join([self._get_speaker_name(spk) for spk in contrib.speakers]) if speakers.strip(): - speaker_word = ngettext(u'Presenter', u'Presenters', len(contrib.speakers)) - speakers = u'- {}: {}'.format(speaker_word, speakers) + speaker_word = ngettext('Presenter', 'Presenters', len(contrib.speakers)) + speakers = f'- {speaker_word}: {speakers}' - text = u'{}{} ({}-{})'.format(escape(contrib.title), room, - to_unicode(format_time(entry.start_dt, timezone=self._tz)), - to_unicode(format_time(entry.end_dt, timezone=self._tz))) - p1 = Paragraph(text.encode('utf-8'), self._styles["session_title"]) + text = '{}{} ({}-{})'.format(escape(contrib.title), room, + format_time(entry.start_dt, timezone=self._tz), + format_time(entry.end_dt, timezone=self._tz)) + p1 = Paragraph(text, self._styles["session_title"]) res.append(p1) if self._ttPDFFormat.showTitleSessionTOC(): - self._indexedFlowable[p1] = {'text': escape(contrib.title.encode('utf-8')), 'level': 2} + self._indexedFlowable[p1] = {'text': escape(contrib.title), 'level': 2} - p2 = Paragraph(speakers.encode('utf-8'), self._styles["conveners"]) + p2 = Paragraph(speakers, self._styles["conveners"]) res.append(p2) if self._ttPDFFormat.showContribAbstract(): - p3 = Paragraph(escape(unicode(contrib.description)), self._styles["contrib_description"]) + p3 = Paragraph(escape(str(contrib.description)), self._styles["contrib_description"]) res.append(p3) if entry == entries[-1]: # if it is the last one, we do the page break and remove the previous one. if self._ttPDFFormat.showNewPagePerSession(): @@ -697,19 +697,19 @@ def _processDayEntries(self, day, story): elif self._ttPDFFormat.showBreaksAtConfLevel() and entry.type == TimetableEntryType.BREAK: break_entry = entry break_ = break_entry.object - room = u'' + room = '' if break_.room_name: - room = u' - {}'.format(escape(break_.room_name)) + room = f' - {escape(break_.room_name)}' - text = u'{}{} ({}-{})'.format(escape(break_.title), room, - to_unicode(format_time(break_entry.start_dt, timezone=self._tz)), - to_unicode(format_time(break_entry.end_dt, timezone=self._tz))) + text = '{}{} ({}-{})'.format(escape(break_.title), room, + format_time(break_entry.start_dt, timezone=self._tz), + format_time(break_entry.end_dt, timezone=self._tz)) - p1 = Paragraph(text.encode('utf-8'), self._styles["session_title"]) + p1 = Paragraph(text, self._styles["session_title"]) res.append(p1) if self._ttPDFFormat.showTitleSessionTOC(): - self._indexedFlowable[p1] = {'text': escape(break_.title.encode('utf-8')), 'level': 2} + self._indexedFlowable[p1] = {'text': escape(break_.title), 'level': 2} if entry == entries[-1]: # if it is the last one, we do the page break and remove the previous one. if self._ttPDFFormat.showNewPagePerSession(): @@ -726,7 +726,7 @@ def getBody(self, story=None): s.fontSize = 18 s.leading = 22 s.alignment = TA_CENTER - p = Paragraph(escape(self.event.title.encode('utf-8')), s) + p = Paragraph(escape(self.event.title), s) story.append(p) story.append(Spacer(1, 0.4 * inch)) @@ -798,7 +798,7 @@ def _defineStyles(self): self._styles["day"] = dayStl def _haveSessionSlotsTitles(self, session): - """Checks if the session has slots with titles or not""" + """Check if the session has slots with titles or not.""" for ss in session.blocks: if ss.title.strip(): return True @@ -827,52 +827,52 @@ def _processDayEntries(self, day, story): lastSessions.append(sess) e = sess title = e.title - res.append(Paragraph(u' {}: {}' - .format(_(u"Session"), escape(title)).encode('utf-8'), + res.append(Paragraph(' {}: {}' + .format(_("Session"), escape(title)), self._styles["normal"])) - room_time = escape(session_slot.room_name) if session_slot.room_name else u'' - room_time = (u' {}: {}({}-{})' - .format(_(u"Time and Place"), room_time, - to_unicode(format_time(entry.start_dt, timezone=self._tz)), - to_unicode(format_time(entry.end_dt, timezone=self._tz)))) + room_time = escape(session_slot.room_name) if session_slot.room_name else '' + room_time = (' {}: {}({}-{})' + .format(_("Time and Place"), room_time, + format_time(entry.start_dt, timezone=self._tz), + format_time(entry.end_dt, timezone=self._tz))) res.append(Paragraph(room_time, self._styles["normal"])) conveners = [c.full_name for c in session_slot.person_links] if conveners: - conveners_text = (u' {}: {}' - .format(ngettext(u'Convener', u'Conveners', len(conveners)), - u'; '.join(conveners))) - res.append(Paragraph(conveners_text.encode('utf-8'), self._styles["normal"])) + conveners_text = (' {}: {}' + .format(ngettext('Convener', 'Conveners', len(conveners)), + '; '.join(conveners))) + res.append(Paragraph(conveners_text, self._styles["normal"])) res.append(Spacer(1, 0.2 * inch)) elif self._ttPDFFormat.showContribsAtConfLevel() and entry.type == TimetableEntryType.CONTRIBUTION: contrib = entry.object if not contrib.can_access(self._user): continue - res.append(Paragraph(u' {}: {}' - .format(_(u"Contribution"), escape(contrib.title)), self._styles["normal"])) + res.append(Paragraph(' {}: {}' + .format(_("Contribution"), escape(contrib.title)), self._styles["normal"])) - room_time = escape(contrib.room_name) if contrib.room_name else u'' - room_time = (u' {}: {}({}-{})' - .format(_(u"Time and Place"), room_time, - to_unicode(format_date(entry.start_dt, timezone=self._tz)), - to_unicode(format_date(entry.end_dt, timezone=self._tz)))) + room_time = escape(contrib.room_name) if contrib.room_name else '' + room_time = (' {}: {}({}-{})' + .format(_("Time and Place"), room_time, + format_date(entry.start_dt, timezone=self._tz), + format_date(entry.end_dt, timezone=self._tz))) res.append(Paragraph(room_time, self._styles["normal"])) spks = [s.full_name for s in contrib.speakers] if spks: - speaker_word = u'{}: '.format(ngettext(u'Presenter', u'Presenters', len(spks))) - res.append(Paragraph(u' {}: {}' - .format(speaker_word, u"; ".join(spks)), self._styles["normal"])) + speaker_word = '{}: '.format(ngettext('Presenter', 'Presenters', len(spks))) + res.append(Paragraph(' {}: {}' + .format(speaker_word, "; ".join(spks)), self._styles["normal"])) res.append(Spacer(1, 0.2 * inch)) elif self._ttPDFFormat.showBreaksAtConfLevel() and entry.type == TimetableEntryType.BREAK: break_ = entry.object title = break_.title - res.append(Paragraph(u' {}: {}' - .format(_(u"Break"), escape(title)), self._styles["normal"])) - room_time = escape(break_.room_name) if break_.room_name else u'' - room_time = (u' {}: {}({}-{})' - .format(_(u"Time and Place"), room_time, - to_unicode(format_date(entry.start_dt, timezone=self._tz)), - to_unicode(format_date(entry.end_dt, timezone=self._tz)))) + res.append(Paragraph(' {}: {}' + .format(_("Break"), escape(title)), self._styles["normal"])) + room_time = escape(break_.room_name) if break_.room_name else '' + room_time = (' {}: {}({}-{})' + .format(_("Time and Place"), room_time, + format_date(entry.start_dt, timezone=self._tz), + format_date(entry.end_dt, timezone=self._tz))) res.append(Paragraph(room_time, self._styles["normal"])) res.append(Spacer(1, 0.2 * inch)) res.append(PageBreak()) @@ -894,16 +894,16 @@ def getBody(self, story=None): if not day_entries: current_day += timedelta(days=1) continue - text = u'{} - {}-{}'.format( + text = '{} - {}-{}'.format( escape(self.event.title), - escape(to_unicode(format_date(self.event.start_dt, timezone=self._tz))), - escape(to_unicode(format_date(self.event.end_dt, timezone=self._tz))) + escape(format_date(self.event.start_dt, timezone=self._tz)), + escape(format_date(self.event.end_dt, timezone=self._tz)) ) if self.event.venue_name: - text = u'%s, %s.' % (text, escape(self.event.venue_name)) - p = Paragraph(text.encode('utf-8'), self._styles["title"]) + text = f'{text}, {escape(self.event.venue_name)}.' + p = Paragraph(text, self._styles["title"]) story.append(p) - text2 = u'{}: {}'.format(_(u'Daily Programme'), escape(current_day.strftime("%A %d %B %Y"))) + text2 = '{}: {}'.format(_('Daily Programme'), escape(current_day.strftime("%A %d %B %Y"))) p2 = Paragraph(text2, self._styles["day"]) story.append(p2) story.append(Spacer(1, 0.4 * inch)) @@ -929,7 +929,7 @@ def firstPage(self, c, doc): c.saveState() c.setFont('Times-Bold', 30) if not self._drawLogo(c): - self._drawWrappedString(c, escape(self.event.title.encode('utf-8')), + self._drawWrappedString(c, escape(self.event.title), height=self._PAGE_HEIGHT - 2*inch) c.setFont('Times-Bold', 25) c.setLineWidth(3) @@ -953,11 +953,11 @@ def _append_text_to_story(text, space=0.2, style=style): story.append(Spacer(inch, space*cm, registration.full_name)) def _print_row(caption, value): - if isinstance(caption, unicode) or is_lazy_string(caption): - caption = unicode(caption).encode('utf-8') - if isinstance(value, unicode) or is_lazy_string(value): - value = unicode(value).encode('utf-8') - text = '{field_name}: {field_value}'.format(field_name=caption, field_value=value) + if isinstance(caption, str) or is_lazy_string(caption): + caption = str(caption) + if isinstance(value, str) or is_lazy_string(value): + value = str(value) + text = f'{caption}: {value}' _append_text_to_story(text) def _print_section(caption): @@ -976,15 +976,15 @@ def _print_section(caption): header_style = ParagraphStyle({}, style, fontSize=16) header_data = [ - [Paragraph(full_name_title.encode('utf-8'), header_style), - Paragraph('#{}'.format(registration.friendly_id), + [Paragraph(full_name_title, header_style), + Paragraph(f'#{registration.friendly_id}', ParagraphStyle({}, header_style, alignment=TA_RIGHT))], ] header_table_style = TableStyle([('LEFTPADDING', (0, 0), (-1, -1), 0), ('RIGHTPADDING', (0, 0), (-1, -1), 0)]) tbl = Table(header_data, style=header_table_style, colWidths=[None, cm]) story.append(tbl) - indexedFlowable[tbl] = {'text': registration.full_name.encode('utf-8'), 'level': 1} + indexedFlowable[tbl] = {'text': registration.full_name, 'level': 1} style = ParagraphStyle({}) style.fontName = 'Sans' @@ -1045,7 +1045,7 @@ def firstPage(self, c, doc): c.saveState() c.setFont('Times-Bold', 30) if not self._drawLogo(c): - self._drawWrappedString(c, escape(self.event.title.encode('utf-8')), + self._drawWrappedString(c, escape(self.event.title), height=self._PAGE_HEIGHT - 2*inch) c.setFont('Times-Bold', 35) c.drawCentredString(self._PAGE_WIDTH/2, self._PAGE_HEIGHT/2, self._title) @@ -1061,13 +1061,11 @@ def laterPages(self, c, doc): c.saveState() c.setFont('Times-Roman', 9) c.setFillColorRGB(0.5, 0.5, 0.5) - confTitle = escape(truncate(self.event.title, 30).encode('utf-8')) + confTitle = escape(truncate(self.event.title, 30)) c.drawString(inch, self._PAGE_HEIGHT - 0.75 * inch, "%s / %s"%(confTitle, self._title)) - title = doc.getCurrentPart() - if len(doc.getCurrentPart())>50: - title = utils.unicodeSlice(doc.getCurrentPart(), 0, 50) + "..." + title = truncate(doc.getCurrentPart(), 50) c.drawRightString(self._PAGE_WIDTH - inch, self._PAGE_HEIGHT - 0.75 * inch, "%s"%title) - c.drawRightString(self._PAGE_WIDTH - inch, 0.75 * inch, u" {} {} ".format(_(u"Page"), doc.page)) + c.drawRightString(self._PAGE_WIDTH - inch, 0.75 * inch, " {} {} ".format(_("Page"), doc.page)) c.drawString(inch, 0.75 * inch, now_utc().strftime("%A %d %B %Y")) c.restoreState() @@ -1103,7 +1101,7 @@ def firstPage(self, c, doc): showLogo = False c.setFont('Times-Bold', 30) if not showLogo: - self._drawWrappedString(c, escape(self.event.title.encode('utf-8')), + self._drawWrappedString(c, escape(self.event.title), height=self._PAGE_HEIGHT - 0.75*inch) c.setFont('Times-Bold', 25) c.setLineWidth(3) @@ -1120,8 +1118,8 @@ def getBody(self, story=None, indexedFlowable={}, level=1 ): style.fontName = "Sans" style.fontSize = 12 style.alignment = TA_CENTER - text = u'{}'.format(_(u"List of registrants")) - p = Paragraph(text, style, part=escape(self.event.title.encode('utf-8'))) + text = '{}'.format(_("List of registrants")) + p = Paragraph(text, style, part=escape(self.event.title)) p.spaceAfter = 30 story.append(p) @@ -1149,11 +1147,11 @@ def getBody(self, story=None, indexedFlowable={}, level=1 ): for item in self._display: if item.input_type == 'accommodation': accommodation_col_counter += 1 - lp.append(Paragraph("{}".format(item.title.encode('utf-8')), text_format)) + lp.append(Paragraph(f"{item.title}", text_format)) lp.append(Paragraph("{}".format(_('Arrival date')), text_format)) lp.append(Paragraph("{}".format(_('Departure date')), text_format)) else: - lp.append(Paragraph("{}".format(item.title.encode('utf-8')), text_format)) + lp.append(Paragraph(f"{item.title}", text_format)) if 'reg_date' in self.static_items: lp.append(Paragraph('{}'.format(_('Registration date')), text_format)) if 'state' in self.static_items: @@ -1169,15 +1167,15 @@ def getBody(self, story=None, indexedFlowable={}, level=1 ): for registration in self._regList: lp = [] lp.append(Paragraph(registration.friendly_id, text_format)) - lp.append(Paragraph("{} {}".format(registration.first_name.encode('utf-8'), - registration.last_name.encode('utf-8')), text_format)) + lp.append(Paragraph("{} {}".format(registration.first_name, + registration.last_name), text_format)) data = registration.data_by_field for item in self._display: friendly_data = data.get(item.id).friendly_data if data.get(item.id) else '' if item.input_type == 'accommodation': if friendly_data: friendly_data = data[item.id].friendly_data - lp.append(Paragraph(friendly_data['choice'].encode('utf-8'), text_format)) + lp.append(Paragraph(friendly_data['choice'], text_format)) lp.append(Paragraph(format_date(friendly_data['arrival_date']), text_format)) lp.append(Paragraph(format_date(friendly_data['departure_date']), text_format)) else: @@ -1187,18 +1185,18 @@ def getBody(self, story=None, indexedFlowable={}, level=1 ): lp.append(Paragraph('', text_format)) elif item.input_type == 'multi_choice': if friendly_data: - multi_choice_data = ', '.join(friendly_data).encode('utf-8') + multi_choice_data = ', '.join(friendly_data) lp.append(Paragraph(multi_choice_data, text_format)) else: lp.append(Paragraph('', text_format)) else: - if isinstance(friendly_data, unicode): - friendly_data = friendly_data.encode('utf-8') + if isinstance(friendly_data, str): + friendly_data = friendly_data lp.append(Paragraph(str(friendly_data), text_format)) if 'reg_date' in self.static_items: lp.append(Paragraph(format_datetime(registration.submitted_dt), text_format)) if 'state' in self.static_items: - lp.append(Paragraph(registration.state.title.encode('utf-8'), text_format)) + lp.append(Paragraph(registration.state.title, text_format)) if 'price' in self.static_items: lp.append(Paragraph(registration.render_price(), text_format)) if 'checked_in' in self.static_items: diff --git a/indico/legacy/pdfinterface/latex.py b/indico/legacy/pdfinterface/latex.py index 0a2a52dd0ab..a666df09b8a 100644 --- a/indico/legacy/pdfinterface/latex.py +++ b/indico/legacy/pdfinterface/latex.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import codecs import os import subprocess @@ -37,10 +35,9 @@ from indico.util.fs import chmod_umask from indico.util.i18n import _, ngettext from indico.util.string import render_markdown -from indico.web.flask.templating import EnsureUnicodeExtension -class PDFLaTeXBase(object): +class PDFLaTeXBase: _table_of_contents = False LATEX_TEMPLATE = None @@ -75,7 +72,7 @@ def generate_source_archive(self): if f.startswith('.') or f.endswith(('.py', '.pyc', '.pyo')): continue file_path = os.path.join(dirpath, f) - archive_name = os.path.relpath(file_path, self.source_dir).encode('utf-8') + archive_name = os.path.relpath(file_path, self.source_dir) zip_handler.write(os.path.abspath(file_path), archive_name) buf.seek(0) return buf @@ -83,17 +80,17 @@ def generate_source_archive(self): class LaTeXRuntimeException(Exception): def __init__(self, source_file, log_file): - super(LaTeXRuntimeException, self).__init__('LaTeX compilation of {} failed'.format(source_file)) + super().__init__(f'LaTeX compilation of {source_file} failed') self.source_file = source_file self.log_file = log_file @property def message(self): - return "Could not compile '{}'. Read '{}' for details".format(self.source_file, self.log_file) + return f"Could not compile '{self.source_file}'. Read '{self.log_file}' for details" class LatexEscapeExtension(Extension): - """Ensures all strings in Jinja are latex-escaped""" + """Ensure all strings in Jinja are latex-escaped.""" def filter_stream(self, stream): in_trans = False @@ -121,20 +118,18 @@ def filter_stream(self, stream): yield token -class RawLatex(unicode): +class RawLatex(str): pass def _latex_escape(s, ignore_braces=False): - if not isinstance(s, basestring) or isinstance(s, RawLatex): + if not isinstance(s, str) or isinstance(s, RawLatex): return s - if isinstance(s, str): - s = s.decode('utf-8') return RawLatex(mdx_latex.latex_escape(s, ignore_braces=ignore_braces)) -class LatexRunner(object): - """Handles the PDF generation from a chosen LaTeX template""" +class LatexRunner: + """Handle the PDF generation from a chosen LaTeX template.""" def __init__(self, source_dir, has_toc=False): self.source_dir = source_dir @@ -152,8 +147,9 @@ def run_latex(self, source_file, log_file=None): subprocess.check_call( pdflatex_cmd, stdout=log_file, + stderr=subprocess.STDOUT, cwd=self.source_dir, - env=dict(os.environ, TEXMFCNF='{}:'.format(os.path.dirname(__file__))) + env=dict(os.environ, TEXMFCNF=f'{os.path.dirname(__file__)}:') ) Logger.get('pdflatex').debug("PDF created successfully!") @@ -178,8 +174,8 @@ def _render_template(self, template_name, kwargs): block_start_string=r'\JINJA{', block_end_string='}', variable_start_string=r'\VAR{', variable_end_string='}', comment_start_string=r'\#{', comment_end_string='}') - env.filters['format_date'] = EnsureUnicodeExtension.wrap_func(format_date) - env.filters['format_time'] = EnsureUnicodeExtension.wrap_func(format_time) + env.filters['format_date'] = format_date + env.filters['format_time'] = format_time env.filters['format_duration'] = lambda delta: format_human_timedelta(delta, 'minutes') env.filters['latex'] = _latex_escape env.filters['rawlatex'] = RawLatex @@ -246,7 +242,7 @@ class AbstractToPDF(PDFLaTeXBase): LATEX_TEMPLATE = 'single_doc' def __init__(self, abstract, tz=None): - super(AbstractToPDF, self).__init__() + super().__init__() self._abstract = abstract event = abstract.event @@ -273,7 +269,7 @@ def _get_track_classification(abstract): return escape(abstract.accepted_track.full_title) else: tracks = sorted(abstract.submitted_for_tracks | abstract.reviewed_for_tracks, key=attrgetter('position')) - return u'; '.join(escape(t.full_title) for t in tracks) + return '; '.join(escape(t.full_title) for t in tracks) @staticmethod def _get_contrib_type(abstract): @@ -285,7 +281,7 @@ class AbstractsToPDF(PDFLaTeXBase): LATEX_TEMPLATE = 'report' def __init__(self, event, abstracts, tz=None): - super(AbstractsToPDF, self).__init__() + super().__init__() if tz is None: self._tz = event.timezone @@ -306,7 +302,7 @@ def __init__(self, event, abstracts, tz=None): class ConfManagerAbstractToPDF(AbstractToPDF): def __init__(self, abstract, tz=None): - super(ConfManagerAbstractToPDF, self).__init__(abstract, tz) + super().__init__(abstract, tz) self._args.update({ 'management': True, @@ -318,18 +314,18 @@ def __init__(self, abstract, tz=None): def _get_status(abstract): state_title = abstract.state.title.upper() if abstract.state == AbstractState.duplicate: - return u"{} (#{}: {})".format(state_title, abstract.duplicate_of.friendly_id, abstract.duplicate_of.title) + return f"{state_title} (#{abstract.duplicate_of.friendly_id}: {abstract.duplicate_of.title})" elif abstract.state == AbstractState.merged: - return u"{} (#{}: {})".format(state_title, abstract.merged_into.friendly_id, abstract.merged_into.title) + return f"{state_title} (#{abstract.merged_into.friendly_id}: {abstract.merged_into.title})" else: return abstract.state.title.upper() @staticmethod def _get_track_reviewing_states(abstract): def _format_review_action(review): - action = unicode(review.proposed_action.title) + action = str(review.proposed_action.title) if review.proposed_action == AbstractAction.accept and review.proposed_contribution_type: - return u'{}: {}'.format(action, review.proposed_contribution_type.name) + return f'{action}: {review.proposed_contribution_type.name}' else: return action @@ -348,19 +344,19 @@ def _format_review_action(review): proposed_contrib_types = {r.proposed_contribution_type.name for r in track_reviews if r.proposed_contribution_type} if proposed_contrib_types: - contrib_types = u', '.join(proposed_contrib_types) - review_state = u'{}: {}'.format(review_state, contrib_types) + contrib_types = ', '.join(proposed_contrib_types) + review_state = f'{review_state}: {contrib_types}' elif track_review_state == AbstractReviewingState.mixed: other_tracks = {x.title for r in track_reviews for x in r.proposed_tracks} proposed_actions = {x.proposed_action for x in track_reviews} no_track_actions = proposed_actions - {AbstractAction.change_tracks} other_info = [] if no_track_actions: - other_info.append(u', '.join(unicode(a.title) for a in no_track_actions)) + other_info.append(', '.join(str(a.title) for a in no_track_actions)) if other_tracks: - other_info.append(_(u"Proposed for other tracks: {}").format(u', '.join(other_tracks))) + other_info.append(_("Proposed for other tracks: {}").format(', '.join(other_tracks))) if other_info: - review_state = u'{}: {}'.format(review_state, u'; '.join(other_info)) + review_state = '{}: {}'.format(review_state, '; '.join(other_info)) elif track_review_state not in {AbstractReviewingState.negative, AbstractReviewingState.conflicting}: continue @@ -370,7 +366,7 @@ def _format_review_action(review): class ConfManagerAbstractsToPDF(AbstractsToPDF): def __init__(self, event, abstracts, tz=None): - super(ConfManagerAbstractsToPDF, self).__init__(event, abstracts, tz) + super().__init__(event, abstracts, tz) self._args.update({ 'management': True, @@ -383,7 +379,7 @@ class ContribToPDF(PDFLaTeXBase): LATEX_TEMPLATE = 'single_doc' def __init__(self, contrib, tz=None): - super(ContribToPDF, self).__init__() + super().__init__() event = contrib.event affiliations, author_mapping, coauthor_mapping = extract_affiliations(contrib) @@ -407,7 +403,7 @@ class ContribsToPDF(PDFLaTeXBase): LATEX_TEMPLATE = 'report' def __init__(self, event, contribs, tz=None): - super(ContribsToPDF, self).__init__() + super().__init__() self._args.update({ 'doc_type': 'contribution', @@ -426,7 +422,7 @@ class ContributionBook(PDFLaTeXBase): LATEX_TEMPLATE = 'contribution_list_book' def __init__(self, event, user, contribs=None, tz=None, sort_by=""): - super(ContributionBook, self).__init__() + super().__init__() tz = tz or event.timezone contribs = sort_contribs(contribs or event.contributions, sort_by) @@ -473,6 +469,6 @@ class AbstractBook(ContributionBook): def __init__(self, event, tz=None): sort_by = boa_settings.get(event, 'sort_by') - super(AbstractBook, self).__init__(event, None, sort_by=sort_by) + super().__init__(event, None, sort_by=sort_by) self._args['show_ids'] = boa_settings.get(event, 'show_abstract_ids') self._args['url'] = None diff --git a/indico/legacy/services/implementation/base.py b/indico/legacy/services/implementation/base.py deleted file mode 100644 index c006231cfc5..00000000000 --- a/indico/legacy/services/implementation/base.py +++ /dev/null @@ -1,54 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from flask import g, session -from werkzeug.exceptions import Forbidden - -from indico.core.logger import sentry_set_tags - - -class ServiceBase(object): - """ - The ServiceBase class is the basic class for services. - """ - - def __init__(self, params): - self._params = params - - def _process_args(self): - pass - - def _check_access(self): - pass - - def process(self): - """ - Processes the request, analyzing the parameters, and feeding them to the - _getAnswer() method (implemented by derived classes) - """ - - g.rh = self - sentry_set_tags({'rh': self.__class__.__name__}) - - self._process_args() - self._check_access() - return self._getAnswer() - - def _getAnswer(self): - """ - To be overloaded. It should contain the code that does the actual - business logic and returns a result (python JSON-serializable object). - If this method is not overloaded, an exception will occur. - If you don't want to return an answer, you should still implement this method with 'pass'. - """ - raise NotImplementedError - - -class LoggedOnlyService(ServiceBase): - def _check_access(self): - if session.user is None: - raise Forbidden("You are currently not authenticated. Please log in again.") diff --git a/indico/legacy/services/implementation/search.py b/indico/legacy/services/implementation/search.py deleted file mode 100644 index a9f9ce38ed5..00000000000 --- a/indico/legacy/services/implementation/search.py +++ /dev/null @@ -1,79 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from __future__ import unicode_literals - -from itertools import chain - -from indico.core.db.sqlalchemy.custom.unaccent import unaccent_match -from indico.legacy.fossils.user import IGroupFossil -from indico.legacy.services.implementation.base import LoggedOnlyService -from indico.modules.events.models.events import Event -from indico.modules.events.models.persons import EventPerson -from indico.modules.events.util import serialize_event_person -from indico.modules.groups import GroupProxy -from indico.modules.users.legacy import search_avatars -from indico.util.fossilize import fossilize -from indico.util.string import sanitize_email, to_unicode - - -class SearchBase(LoggedOnlyService): - def _process_args(self): - self._searchExt = self._params.get('search-ext', False) - - -class SearchUsers(SearchBase): - def _process_args(self): - SearchBase._process_args(self) - self._surName = self._params.get("surName", "") - self._name = self._params.get("name", "") - self._organisation = self._params.get("organisation", "") - self._email = sanitize_email(self._params.get("email", "")) - self._exactMatch = self._params.get("exactMatch", False) - self._confId = self._params.get("conferenceId", None) - self._event = Event.get(self._confId, is_deleted=False) if self._confId else None - - def _getAnswer(self): - event_persons = [] - criteria = { - 'surName': self._surName, - 'name': self._name, - 'organisation': self._organisation, - 'email': self._email - } - users = search_avatars(criteria, self._exactMatch, self._searchExt) - if self._event: - fields = {EventPerson.first_name: self._name, - EventPerson.last_name: self._surName, - EventPerson.email: self._email, - EventPerson.affiliation: self._organisation} - criteria = [unaccent_match(col, val, exact=self._exactMatch) for col, val in fields.iteritems()] - event_persons = self._event.persons.filter(*criteria).all() - fossilized_users = fossilize(sorted(users, key=lambda av: (av.getStraightFullName(), av.getEmail()))) - fossilized_event_persons = map(serialize_event_person, event_persons) - unique_users = {to_unicode(user['email']): user for user in chain(fossilized_users, fossilized_event_persons)} - return sorted(unique_users.values(), key=lambda x: (to_unicode(x['name']).lower(), to_unicode(x['email']))) - - -class SearchGroups(SearchBase): - def _process_args(self): - SearchBase._process_args(self) - self._group = self._params.get("group", "").strip() - self._exactMatch = self._params.get("exactMatch", False) - - def _getAnswer(self): - results = [g.as_legacy_group for g in GroupProxy.search(self._group, exact=self._exactMatch)] - fossilized_results = fossilize(results, IGroupFossil) - for fossilizedGroup in fossilized_results: - fossilizedGroup["isGroup"] = True - return fossilized_results - - -methodMap = { - "users": SearchUsers, - "groups": SearchGroups, -} diff --git a/indico/legacy/services/interface/rpc/__init__.py b/indico/legacy/services/interface/rpc/__init__.py deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/indico/legacy/services/interface/rpc/json.py b/indico/legacy/services/interface/rpc/json.py deleted file mode 100644 index bd0d8612b7d..00000000000 --- a/indico/legacy/services/interface/rpc/json.py +++ /dev/null @@ -1,23 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from flask import jsonify, request - -from indico.core.logger import Logger -from indico.legacy.services.interface.rpc.process import invoke_method -from indico.util.fossilize import clearCache - - -def process(): - clearCache() - payload = request.json - Logger.get('rpc').info('json rpc request. request: %r', payload) - rv = {} - if 'id' in payload: - rv['id'] = payload['id'] - rv['result'] = invoke_method(str(payload['method']), payload.get('params', [])) - return jsonify(rv) diff --git a/indico/legacy/services/interface/rpc/process.py b/indico/legacy/services/interface/rpc/process.py deleted file mode 100644 index 8e97222617d..00000000000 --- a/indico/legacy/services/interface/rpc/process.py +++ /dev/null @@ -1,72 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -import copy - -from flask import request, session -from sqlalchemy.exc import DatabaseError -from werkzeug.exceptions import BadRequest - -from indico.core import signals -from indico.core.db import db -from indico.core.db.sqlalchemy.core import handle_sqlalchemy_database_error -from indico.core.notifications import flush_email_queue, init_email_queue -from indico.legacy.services.interface.rpc import handlers -from indico.util import fossilize -from indico.util.i18n import _ - - -def _lookup_handler(method): - endpoint, functionName = handlers, method - while True: - handler = endpoint.methodMap.get(functionName, None) - if handler: - break - try: - endpointName, functionName = method.split('.', 1) - except Exception: - raise BadRequest('Unsupported method') - - if 'endpointMap' in dir(endpoint): - endpoint = endpoint.endpointMap.get(endpointName, None) - if not endpoint: - raise BadRequest('Unknown endpoint: {}'.format(endpointName)) - else: - raise BadRequest('Unsupported method') - return handler - - -def _process_request(method, params): - handler = _lookup_handler(method) - - if session.csrf_protected and session.csrf_token != request.headers.get('X-CSRF-Token'): - msg = _(u"It looks like there was a problem with your current session. Please use your browser's back " - u"button, reload the page and try again.") - raise BadRequest(msg) - - if hasattr(handler, 'process'): - return handler(params).process() - else: - return handler(params) - - -def invoke_method(method, params): - result = None - fossilize.clearCache() - init_email_queue() - try: - result = _process_request(method, copy.deepcopy(params)) - signals.after_process.send() - db.session.commit() - except DatabaseError: - db.session.rollback() - handle_sqlalchemy_database_error() - except Exception: - db.session.rollback() - raise - flush_email_queue() - return result diff --git a/indico/legacy/services/tools.py b/indico/legacy/services/tools.py deleted file mode 100644 index 87506231d97..00000000000 --- a/indico/legacy/services/tools.py +++ /dev/null @@ -1,15 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - - -def toJsDate(datetime): - return "(new Date(%i,%i,%i,%i,%i,%i))" % (datetime.year, - datetime.month - 1, - datetime.day, - datetime.hour, - datetime.minute, - datetime.second) diff --git a/indico/legacy/webinterface/pages/static.py b/indico/legacy/webinterface/pages/static.py index 0b5bf598919..2a65f4cf722 100644 --- a/indico/legacy/webinterface/pages/static.py +++ b/indico/legacy/webinterface/pages/static.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -16,10 +16,10 @@ class WPStaticEventBase: def _get_header(self): - return u"" + return "" def _get_footer(self): - return u"" + return "" class WPStaticSimpleEventDisplay(WPStaticEventBase, WPSimpleEventDisplay): diff --git a/indico/migrations/env.py b/indico/migrations/env.py index 17ef64f35f6..5771763edb9 100644 --- a/indico/migrations/env.py +++ b/indico/migrations/env.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -26,11 +26,12 @@ target_metadata = current_app.extensions['migrate'].db.metadata -def _include_symbol(tablename, schema): - # We ignore plugin tables in migrations - if schema and schema.startswith('plugin_'): +def _include_object(object_, name, type_, reflected, compare_to): + if type_ != 'table': + return True + if object_.schema and object_.schema.startswith('plugin_'): return False - return tablename != 'alembic_version' and not tablename.startswith('alembic_version_') + return name != 'alembic_version' and not name.startswith('alembic_version_') def _render_item(type_, obj, autogen_context): @@ -52,11 +53,10 @@ def run_migrations_offline(): Calls to context.execute() here emit the given string to the script output. - """ url = config.get_main_option('sqlalchemy.url') context.configure(url=url, target_metadata=target_metadata, include_schemas=True, - version_table_schema='public', include_symbol=_include_symbol, render_item=_render_item, + version_table_schema='public', include_object=_include_object, render_item=_render_item, template_args={'toplevel_code': set()}) with context.begin_transaction(): @@ -68,7 +68,6 @@ def run_migrations_online(): In this scenario we need to create an Engine and associate a connection with the context. - """ engine = engine_from_config(config.get_section(config.config_ini_section), prefix='sqlalchemy.', @@ -76,7 +75,7 @@ def run_migrations_online(): connection = engine.connect() context.configure(connection=connection, target_metadata=target_metadata, include_schemas=True, - version_table_schema='public', include_symbol=_include_symbol, render_item=_render_item, + version_table_schema='public', include_object=_include_object, render_item=_render_item, template_args={'toplevel_code': set()}) try: diff --git a/indico/migrations/versions/20171124_1138_2af245be72a6_review_questions_models.py b/indico/migrations/versions/20171124_1138_2af245be72a6_review_questions_models.py index 97f98dac188..6bd52222a38 100644 --- a/indico/migrations/versions/20171124_1138_2af245be72a6_review_questions_models.py +++ b/indico/migrations/versions/20171124_1138_2af245be72a6_review_questions_models.py @@ -34,8 +34,8 @@ def upgrade(): op.add_column(questions_table, sa.Column('description', sa.Text(), nullable=False, server_default=''), schema=schema) op.alter_column(questions_table, 'description', server_default=None, schema=schema) - op.execute('ALTER TABLE {0}.{1} ALTER COLUMN "value" TYPE JSON USING to_json(value)'.format(schema, - ratings_table)) + op.execute('ALTER TABLE {}.{} ALTER COLUMN "value" TYPE JSON USING to_json(value)'.format(schema, + ratings_table)) def downgrade(): @@ -43,9 +43,9 @@ def downgrade(): op.alter_column(questions_table, 'title', new_column_name='text', schema=schema) op.execute("DELETE FROM {0}.{1} WHERE question_id IN(SELECT id FROM {0}.{2} " "WHERE field_type != 'rating' OR NOT is_required)".format(schema, ratings_table, questions_table)) - op.execute("DELETE FROM {0}.{1} WHERE field_type != 'rating'".format(schema, questions_table)) - op.execute('ALTER TABLE {0}.{1} ALTER COLUMN "value" TYPE INT USING value::TEXT::INT'.format(schema, - ratings_table)) + op.execute(f"DELETE FROM {schema}.{questions_table} WHERE field_type != 'rating'") + op.execute('ALTER TABLE {}.{} ALTER COLUMN "value" TYPE INT USING value::TEXT::INT'.format(schema, + ratings_table)) op.drop_column(questions_table, 'field_type', schema=schema) op.drop_column(questions_table, 'is_required', schema=schema) diff --git a/indico/migrations/versions/20181023_1209_7a72d63acba9_update_map_aspects_structure.py b/indico/migrations/versions/20181023_1209_7a72d63acba9_update_map_aspects_structure.py index 09d945a8f8c..33a8119d6a1 100644 --- a/indico/migrations/versions/20181023_1209_7a72d63acba9_update_map_aspects_structure.py +++ b/indico/migrations/versions/20181023_1209_7a72d63acba9_update_map_aspects_structure.py @@ -51,7 +51,7 @@ def downgrade(): if default_location_id is None: # We have some aspects that cannot be associated with a location since there is none conn.execute('DELETE FROM roombooking.aspects') - default_location = unicode(default_location_id) if default_location_id is not None else None + default_location = str(default_location_id) if default_location_id is not None else None default_aspect_id = conn.execute('SELECT id FROM roombooking.aspects WHERE is_default').scalar() op.add_column('locations', sa.Column('default_aspect_id', sa.Integer, nullable=True), schema='roombooking') op.create_foreign_key(op.f('fk_locations_default_aspect_id'), 'locations', 'aspects', diff --git a/indico/migrations/versions/20181024_1424_27c45c384d65_make_equipment_types_global.py b/indico/migrations/versions/20181024_1424_27c45c384d65_make_equipment_types_global.py index a0c6daadee9..c51d5cbd48f 100644 --- a/indico/migrations/versions/20181024_1424_27c45c384d65_make_equipment_types_global.py +++ b/indico/migrations/versions/20181024_1424_27c45c384d65_make_equipment_types_global.py @@ -35,7 +35,7 @@ def _make_names_unique(): conflict = conn.execute("SELECT COUNT(*) FROM roombooking.equipment_types WHERE id != %s AND name = %s", (row.id, row.name)).scalar() if conflict: - new_name = '{} ({})'.format(row.name, row.location) + new_name = f'{row.name} ({row.location})' conn.execute('UPDATE roombooking.equipment_types SET name = %s WHERE id = %s', (new_name, row.id)) @@ -57,7 +57,7 @@ def downgrade(): if default_location_id is None: if conn.execute('SELECT COUNT(*) FROM roombooking.locations').scalar(): raise Exception('Please set a default location') - default_location = unicode(default_location_id) if default_location_id is not None else None + default_location = str(default_location_id) if default_location_id is not None else None op.add_column('equipment_types', sa.Column('location_id', sa.Integer(), nullable=False, server_default=default_location), schema='roombooking') diff --git a/indico/migrations/versions/20181025_1148_ec410be271df_make_room_attributes_global.py b/indico/migrations/versions/20181025_1148_ec410be271df_make_room_attributes_global.py index 2eb7a7e8f46..00f5f31d6e2 100644 --- a/indico/migrations/versions/20181025_1148_ec410be271df_make_room_attributes_global.py +++ b/indico/migrations/versions/20181025_1148_ec410be271df_make_room_attributes_global.py @@ -35,7 +35,7 @@ def _make_names_unique(): conflict = conn.execute("SELECT COUNT(*) FROM roombooking.room_attributes WHERE id != %s AND name = %s", (row.id, row.name)).scalar() if conflict: - new_name = '{} ({})'.format(row.name, row.location) + new_name = f'{row.name} ({row.location})' conn.execute('UPDATE roombooking.room_attributes SET name = %s WHERE id = %s', (new_name, row.id)) @@ -58,7 +58,7 @@ def downgrade(): if default_location_id is None: if conn.execute('SELECT COUNT(*) FROM roombooking.locations').scalar(): raise Exception('Please set a default location') - default_location = unicode(default_location_id) if default_location_id is not None else None + default_location = str(default_location_id) if default_location_id is not None else None op.add_column('room_attributes', sa.Column('location_id', sa.Integer(), nullable=False, server_default=default_location), schema='roombooking') diff --git a/indico/migrations/versions/20181213_1110_cbe630695800_add_room_principals_table.py b/indico/migrations/versions/20181213_1110_cbe630695800_add_room_principals_table.py index c99e92e605a..8a00d8c2091 100644 --- a/indico/migrations/versions/20181213_1110_cbe630695800_add_room_principals_table.py +++ b/indico/migrations/versions/20181213_1110_cbe630695800_add_room_principals_table.py @@ -5,7 +5,6 @@ Create Date: 2018-12-13 11:10:12.684382 """ -from __future__ import print_function import json @@ -96,14 +95,14 @@ def _upgrade_permissions(): if booking_group: group_kwargs = _group_to_kwargs(booking_group) if group_kwargs is None: - print('WARNING: Invalid booking group: {}'.format(booking_group)) + print(f'WARNING: Invalid booking group: {booking_group}') else: permission = 'prebook' if reservations_need_confirmation else 'book' _create_acl_entry(conn, room_id, permissions={permission}, **group_kwargs) if manager_group: group_kwargs = _group_to_kwargs(manager_group) if group_kwargs is None: - print('WARNING: Invalid manager group: {}'.format(manager_group)) + print(f'WARNING: Invalid manager group: {manager_group}') else: _create_acl_entry(conn, room_id, full_access=True, **group_kwargs) @@ -161,9 +160,9 @@ def _downgrade_permissions(): continue if row.type == PrincipalType.local_group and not default_group_provider: if row.full_access: - _set_attribute_value(conn, room_id, manager_group_attr_id, unicode(row.local_group_id)) + _set_attribute_value(conn, room_id, manager_group_attr_id, str(row.local_group_id)) if 'book' in row.permissions or 'prebook' in row.permissions: - _set_attribute_value(conn, room_id, booking_group_attr_id, unicode(row.local_group_id)) + _set_attribute_value(conn, room_id, booking_group_attr_id, str(row.local_group_id)) elif (row.type == PrincipalType.multipass_group and default_group_provider and row.mp_group_provider == default_group_provider.name): if row.full_access: @@ -212,7 +211,7 @@ def upgrade(): postgresql_where=sa.text('type = 3')) op.add_column('rooms', sa.Column('protection_mode', PyIntEnum(ProtectionMode, exclude_values={ProtectionMode.inheriting}), - nullable=False, server_default=unicode(ProtectionMode.protected.value)), + nullable=False, server_default=str(ProtectionMode.protected.value)), schema='roombooking') _upgrade_permissions() op.alter_column('rooms', 'protection_mode', server_default=None, schema='roombooking') diff --git a/indico/migrations/versions/20200324_1004_a3295d628e3b_migrate_event_labels_from_settings.py b/indico/migrations/versions/20200324_1004_a3295d628e3b_migrate_event_labels_from_settings.py index f536ae72433..29ed3b2b5be 100644 --- a/indico/migrations/versions/20200324_1004_a3295d628e3b_migrate_event_labels_from_settings.py +++ b/indico/migrations/versions/20200324_1004_a3295d628e3b_migrate_event_labels_from_settings.py @@ -60,7 +60,7 @@ def downgrade(): mapping = {} res = conn.execute("SELECT id, title, color FROM events.labels") for label_id, title, color in res: - uuid = unicode(uuid4()) + uuid = str(uuid4()) conn.execute("INSERT INTO indico.settings (module, name, value) VALUES ('event_labels', %s, %s)", (uuid, json.dumps({'id': uuid, 'title': title, 'color': color}))) mapping[label_id] = uuid diff --git a/indico/migrations/versions/20200402_1113_933665578547_migrate_review_conditions_from_settings.py b/indico/migrations/versions/20200402_1113_933665578547_migrate_review_conditions_from_settings.py index 878f463d0c2..d4642733cfc 100644 --- a/indico/migrations/versions/20200402_1113_933665578547_migrate_review_conditions_from_settings.py +++ b/indico/migrations/versions/20200402_1113_933665578547_migrate_review_conditions_from_settings.py @@ -28,7 +28,7 @@ def upgrade(): for type_ in EditableType: res = conn.execute( "SELECT event_id, value FROM events.settings WHERE module = 'editing' AND name = %s", - ("{}_review_conditions".format(type_.name),), + (f"{type_.name}_review_conditions",), ) for event_id, value in res: for condition in value: @@ -45,7 +45,7 @@ def upgrade(): ) conn.execute( "DELETE FROM events.settings WHERE module = 'editing' AND name = %s", - ("{}_review_conditions".format(type_.name),), + (f"{type_.name}_review_conditions",), ) @@ -61,12 +61,12 @@ def downgrade(): "SELECT file_type_id FROM event_editing.review_condition_file_types WHERE review_condition_id = %s", (id,), ) - value = [unicode(uuid4()), [f[0] for f in file_types.fetchall()]] + value = [str(uuid4()), [f[0] for f in file_types.fetchall()]] review_conditions[event_id].append(value) for key, value in review_conditions.items(): conn.execute( "INSERT INTO events.settings (event_id, module, name, value) VALUES (%s, 'editing', %s, %s)", - (key, "{}_review_conditions".format(type_.name), json.dumps(value)), + (key, f"{type_.name}_review_conditions", json.dumps(value)), ) conn.execute("DELETE FROM event_editing.review_condition_file_types") diff --git a/indico/migrations/versions/20200615_2142_c0fc1e46888b_disallow_editing_permissions_for_groups.py b/indico/migrations/versions/20200615_2142_c0fc1e46888b_disallow_editing_permissions_for_groups.py index 78698d9e524..123331d9f35 100644 --- a/indico/migrations/versions/20200615_2142_c0fc1e46888b_disallow_editing_permissions_for_groups.py +++ b/indico/migrations/versions/20200615_2142_c0fc1e46888b_disallow_editing_permissions_for_groups.py @@ -17,7 +17,7 @@ def upgrade(): permissions = "ARRAY['paper_editing', 'slides_editing', 'poster_editing']" - condition = 'type NOT IN (2, 3) OR (NOT (permissions::text[] && {}))'.format(permissions) + condition = f'type NOT IN (2, 3) OR (NOT (permissions::text[] && {permissions}))' op.create_check_constraint('disallow_group_editor_permissions', 'principals', condition, schema='events') diff --git a/indico/migrations/versions/20201103_1431_8d614ef75968_allow_mx_user_title.py b/indico/migrations/versions/20201103_1431_8d614ef75968_allow_mx_user_title.py new file mode 100644 index 00000000000..4938b71ef66 --- /dev/null +++ b/indico/migrations/versions/20201103_1431_8d614ef75968_allow_mx_user_title.py @@ -0,0 +1,62 @@ +"""Allow mx user title + +Revision ID: 8d614ef75968 +Revises: f37d509e221c +Create Date: 2020-11-03 14:31:44.639447 +""" + +from alembic import op + + +# revision identifiers, used by Alembic. +revision = '8d614ef75968' +down_revision = 'f37d509e221c' +branch_labels = None +depends_on = None + + +def upgrade(): + op.execute(''' + ALTER TABLE "event_abstracts"."abstract_person_links" DROP CONSTRAINT "ck_abstract_person_links_valid_enum_title"; + ALTER TABLE "events"."contribution_person_links" DROP CONSTRAINT "ck_contribution_person_links_valid_enum_title"; + ALTER TABLE "events"."event_person_links" DROP CONSTRAINT "ck_event_person_links_valid_enum_title"; + ALTER TABLE "events"."persons" DROP CONSTRAINT "ck_persons_valid_enum_title"; + ALTER TABLE "events"."session_block_person_links" DROP CONSTRAINT "ck_session_block_person_links_valid_enum_title"; + ALTER TABLE "events"."subcontribution_person_links" DROP CONSTRAINT "ck_subcontribution_person_links_valid_enum_title"; + ALTER TABLE "users"."users" DROP CONSTRAINT "ck_users_valid_enum_title"; + ALTER TABLE "event_abstracts"."abstract_person_links" ADD CONSTRAINT "ck_abstract_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ALTER TABLE "events"."contribution_person_links" ADD CONSTRAINT "ck_contribution_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ALTER TABLE "events"."event_person_links" ADD CONSTRAINT "ck_event_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ALTER TABLE "events"."persons" ADD CONSTRAINT "ck_persons_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ALTER TABLE "events"."session_block_person_links" ADD CONSTRAINT "ck_session_block_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ALTER TABLE "events"."subcontribution_person_links" ADD CONSTRAINT "ck_subcontribution_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ALTER TABLE "users"."users" ADD CONSTRAINT "ck_users_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5, 6]))); + ''') + + +def downgrade(): + op.execute(''' + UPDATE "event_abstracts"."abstract_person_links" SET title = 0 WHERE title = 6; + UPDATE "events"."contribution_person_links" SET title = 0 WHERE title = 6; + UPDATE "events"."event_person_links" SET title = 0 WHERE title = 6; + UPDATE "events"."persons" SET title = 0 WHERE title = 6; + UPDATE "events"."session_block_person_links" SET title = 0 WHERE title = 6; + UPDATE "events"."subcontribution_person_links" SET title = 0 WHERE title = 6; + UPDATE "users"."users" SET title = 0 WHERE title = 6; + ''') + op.execute(''' + ALTER TABLE "event_abstracts"."abstract_person_links" DROP CONSTRAINT "ck_abstract_person_links_valid_enum_title"; + ALTER TABLE "events"."contribution_person_links" DROP CONSTRAINT "ck_contribution_person_links_valid_enum_title"; + ALTER TABLE "events"."event_person_links" DROP CONSTRAINT "ck_event_person_links_valid_enum_title"; + ALTER TABLE "events"."persons" DROP CONSTRAINT "ck_persons_valid_enum_title"; + ALTER TABLE "events"."session_block_person_links" DROP CONSTRAINT "ck_session_block_person_links_valid_enum_title"; + ALTER TABLE "events"."subcontribution_person_links" DROP CONSTRAINT "ck_subcontribution_person_links_valid_enum_title"; + ALTER TABLE "users"."users" DROP CONSTRAINT "ck_users_valid_enum_title"; + ALTER TABLE "event_abstracts"."abstract_person_links" ADD CONSTRAINT "ck_abstract_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ALTER TABLE "events"."contribution_person_links" ADD CONSTRAINT "ck_contribution_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ALTER TABLE "events"."event_person_links" ADD CONSTRAINT "ck_event_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ALTER TABLE "events"."persons" ADD CONSTRAINT "ck_persons_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ALTER TABLE "events"."session_block_person_links" ADD CONSTRAINT "ck_session_block_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ALTER TABLE "events"."subcontribution_person_links" ADD CONSTRAINT "ck_subcontribution_person_links_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ALTER TABLE "users"."users" ADD CONSTRAINT "ck_users_valid_enum_title" CHECK ((title = ANY (ARRAY[0, 1, 2, 3, 4, 5]))); + ''') diff --git a/indico/migrations/versions/20201209_2010_e4fb983dc64c_add_until_approved_regform_modification_mode.py b/indico/migrations/versions/20201209_2010_e4fb983dc64c_add_until_approved_regform_modification_mode.py new file mode 100644 index 00000000000..fbe8f1561fc --- /dev/null +++ b/indico/migrations/versions/20201209_2010_e4fb983dc64c_add_until_approved_regform_modification_mode.py @@ -0,0 +1,30 @@ +"""Add 'until approved' regform modification mode + +Revision ID: e4fb983dc64c +Revises: 8d614ef75968 +Create Date: 2020-12-09 20:10:12.155982 +""" + +from alembic import op + + +# revision identifiers, used by Alembic. +revision = 'e4fb983dc64c' +down_revision = '8d614ef75968' +branch_labels = None +depends_on = None + + +def upgrade(): + op.execute(''' + ALTER TABLE "event_registration"."forms" DROP CONSTRAINT "ck_forms_valid_enum_modification_mode"; + ALTER TABLE "event_registration"."forms" ADD CONSTRAINT "ck_forms_valid_enum_modification_mode" CHECK ((modification_mode = ANY (ARRAY[1, 2, 3, 4]))); + ''') + + +def downgrade(): + op.execute(''' + UPDATE "event_registration"."forms" SET modification_mode = 3 WHERE modification_mode = 4; + ALTER TABLE "event_registration"."forms" DROP CONSTRAINT "ck_forms_valid_enum_modification_mode"; + ALTER TABLE "event_registration"."forms" ADD CONSTRAINT "ck_forms_valid_enum_modification_mode" CHECK ((modification_mode = ANY (ARRAY[1, 2, 3]))); + ''') diff --git a/indico/migrations/versions/20210129_2232_e787389ca868_add_rejection_reason.py b/indico/migrations/versions/20210129_2232_e787389ca868_add_rejection_reason.py new file mode 100644 index 00000000000..69b00ff8945 --- /dev/null +++ b/indico/migrations/versions/20210129_2232_e787389ca868_add_rejection_reason.py @@ -0,0 +1,26 @@ +"""add 'rejection_reason' to registration. + +Revision ID: e787389ca868 +Revises: e4fb983dc64c +Create Date: 2021-01-29 22:32:11.206740 +""" + +import sqlalchemy as sa +from alembic import op + + +# revision identifiers, used by Alembic. +revision = 'e787389ca868' +down_revision = 'e4fb983dc64c' +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column('registrations', sa.Column('rejection_reason', sa.String(), server_default='', nullable=False), + schema='event_registration') + op.alter_column('registrations', 'rejection_reason', server_default=None, schema='event_registration') + + +def downgrade(): + op.drop_column('registrations', 'rejection_reason', schema='event_registration') diff --git a/indico/migrations/versions/20210211_1613_26985db8ed12_add_attach_ical_to_reminders.py b/indico/migrations/versions/20210211_1613_26985db8ed12_add_attach_ical_to_reminders.py new file mode 100644 index 00000000000..4030ea735af --- /dev/null +++ b/indico/migrations/versions/20210211_1613_26985db8ed12_add_attach_ical_to_reminders.py @@ -0,0 +1,26 @@ +"""Add attach_ical to reminders + +Revision ID: 26985db8ed12 +Revises: e787389ca868 +Create Date: 2021-02-11 16:13:25.061874 +""" + +import sqlalchemy as sa +from alembic import op + + +# revision identifiers, used by Alembic. +revision = '26985db8ed12' +down_revision = 'e787389ca868' +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column('reminders', sa.Column('attach_ical', sa.Boolean(), nullable=False, + server_default='false'), schema='events') + op.alter_column('reminders', 'attach_ical', server_default=None, schema='events') + + +def downgrade(): + op.drop_column('reminders', 'attach_ical', schema='events') diff --git a/indico/migrations/versions/20210215_1052_f26c201c8254_add_attach_ical_to_registrationform.py b/indico/migrations/versions/20210215_1052_f26c201c8254_add_attach_ical_to_registrationform.py new file mode 100644 index 00000000000..b6e2565c192 --- /dev/null +++ b/indico/migrations/versions/20210215_1052_f26c201c8254_add_attach_ical_to_registrationform.py @@ -0,0 +1,26 @@ +"""Add attach_ical to RegistrationForm + +Revision ID: f26c201c8254 +Revises: 26985db8ed12 +Create Date: 2021-02-15 10:52:15.353452 +""" + +import sqlalchemy as sa +from alembic import op + + +# revision identifiers, used by Alembic. +revision = 'f26c201c8254' +down_revision = '26985db8ed12' +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column('forms', sa.Column('attach_ical', sa.Boolean(), nullable=False, + server_default='false'), schema='event_registration') + op.alter_column('forms', 'attach_ical', server_default=None, schema='event_registration') + + +def downgrade(): + op.drop_column('forms', 'attach_ical', schema='event_registration') diff --git a/indico/migrations/versions/20210219_1428_3782de7970da_rename_oauth_default_scopes.py b/indico/migrations/versions/20210219_1428_3782de7970da_rename_oauth_default_scopes.py new file mode 100644 index 00000000000..257df8848f1 --- /dev/null +++ b/indico/migrations/versions/20210219_1428_3782de7970da_rename_oauth_default_scopes.py @@ -0,0 +1,23 @@ +"""Rename oauth default_scopes + +Revision ID: 3782de7970da +Revises: f26c201c8254 +Create Date: 2021-02-19 14:28:23.469730 +""" + +from alembic import op + + +# revision identifiers, used by Alembic. +revision = '3782de7970da' +down_revision = 'f26c201c8254' +branch_labels = None +depends_on = None + + +def upgrade(): + op.alter_column('applications', 'default_scopes', new_column_name='allowed_scopes', schema='oauth') + + +def downgrade(): + op.alter_column('applications', 'allowed_scopes', new_column_name='default_scopes', schema='oauth') diff --git a/indico/migrations/versions/20210219_1555_da06d8f50342_separate_authorized_scopes_from_tokens.py b/indico/migrations/versions/20210219_1555_da06d8f50342_separate_authorized_scopes_from_tokens.py new file mode 100644 index 00000000000..caf5b15491a --- /dev/null +++ b/indico/migrations/versions/20210219_1555_da06d8f50342_separate_authorized_scopes_from_tokens.py @@ -0,0 +1,77 @@ +"""Separate authorized scopes from tokens + +Revision ID: da06d8f50342 +Revises: 3782de7970da +Create Date: 2021-02-19 15:55:53.134744 +""" + +import sqlalchemy as sa +from alembic import op +from sqlalchemy.dialects import postgresql + + +# revision identifiers, used by Alembic. +revision = 'da06d8f50342' +down_revision = '3782de7970da' +branch_labels = None +depends_on = None + + +def upgrade(): + op.create_table( + 'application_user_links', + sa.Column('id', sa.Integer(), nullable=False, primary_key=True), + sa.Column('application_id', sa.Integer(), nullable=False, index=True), + sa.Column('user_id', sa.Integer(), nullable=False, index=True), + sa.Column('scopes', postgresql.ARRAY(sa.String()), nullable=False), + sa.ForeignKeyConstraint(['application_id'], ['oauth.applications.id']), + sa.ForeignKeyConstraint(['user_id'], ['users.users.id']), + sa.UniqueConstraint('application_id', 'user_id'), + schema='oauth' + ) + op.create_unique_constraint(None, 'tokens', ['application_id', 'user_id', 'scopes'], schema='oauth') + op.drop_constraint('uq_tokens_application_id_user_id', 'tokens', schema='oauth') + op.execute(''' + INSERT INTO oauth.application_user_links (application_id, user_id, scopes) + SELECT application_id, user_id, scopes FROM oauth.tokens; + ''') + op.add_column('tokens', sa.Column('app_user_link_id', sa.Integer(), nullable=True), schema='oauth') + op.create_index(None, 'tokens', ['app_user_link_id'], unique=False, schema='oauth') + op.create_unique_constraint(None, 'tokens', ['app_user_link_id', 'scopes'], schema='oauth') + op.execute(''' + UPDATE oauth.tokens t SET app_user_link_id = ( + SELECT id FROM oauth.application_user_links WHERE application_id = t.application_id AND user_id = t.user_id + ); + ''') + op.alter_column('tokens', 'app_user_link_id', nullable=False, schema='oauth') + op.create_foreign_key(None, 'tokens', 'application_user_links', ['app_user_link_id'], ['id'], + source_schema='oauth', referent_schema='oauth', ondelete='CASCADE') + op.drop_column('tokens', 'application_id', schema='oauth') + op.drop_column('tokens', 'user_id', schema='oauth') + + +def downgrade(): + op.add_column('tokens', sa.Column('application_id', sa.Integer(), nullable=True), schema='oauth') + op.add_column('tokens', sa.Column('user_id', sa.Integer(), nullable=True), schema='oauth') + op.create_index(None, 'tokens', ['user_id'], unique=False, schema='oauth') + op.execute(''' + DELETE FROM oauth.tokens + WHERE app_user_link_id IN ( + SELECT app_user_link_id FROM oauth.tokens GROUP BY app_user_link_id HAVING COUNT(*) > 1 + ); + + UPDATE oauth.tokens t SET application_id = ( + SELECT application_id FROM oauth.application_user_links WHERE id = t.app_user_link_id + ), user_id = ( + SELECT user_id FROM oauth.application_user_links WHERE id = t.app_user_link_id + ); + ''') + op.create_foreign_key(None, 'tokens', 'applications', ['application_id'], ['id'], + source_schema='oauth', referent_schema='oauth') + op.create_foreign_key(None, 'tokens', 'users', ['user_id'], ['id'], + source_schema='oauth', referent_schema='users') + op.alter_column('tokens', 'application_id', nullable=False, schema='oauth') + op.alter_column('tokens', 'user_id', nullable=False, schema='oauth') + op.drop_column('tokens', 'app_user_link_id', schema='oauth') + op.create_unique_constraint(None, 'tokens', ['application_id', 'user_id'], schema='oauth') + op.drop_table('application_user_links', schema='oauth') diff --git a/indico/migrations/versions/20210222_1754_c36abe1c23c7_make_oauth_pkce_flow_configurable.py b/indico/migrations/versions/20210222_1754_c36abe1c23c7_make_oauth_pkce_flow_configurable.py new file mode 100644 index 00000000000..19449deca7c --- /dev/null +++ b/indico/migrations/versions/20210222_1754_c36abe1c23c7_make_oauth_pkce_flow_configurable.py @@ -0,0 +1,27 @@ +"""Make OAuth PKCE flow configurable + +Revision ID: c36abe1c23c7 +Revises: da06d8f50342 +Create Date: 2021-02-22 17:54:16.270477 +""" + +import sqlalchemy as sa +from alembic import op + + +# revision identifiers, used by Alembic. +revision = 'c36abe1c23c7' +down_revision = 'da06d8f50342' +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column('applications', sa.Column('allow_pkce_flow', sa.Boolean(), server_default='false', nullable=False), + schema='oauth') + op.execute('UPDATE oauth.applications SET allow_pkce_flow = true WHERE system_app_type = 1') + op.alter_column('applications', 'allow_pkce_flow', server_default=None, schema='oauth') + + +def downgrade(): + op.drop_column('applications', 'allow_pkce_flow', schema='oauth') diff --git a/indico/migrations/versions/20210222_1914_d354278c6d95_store_tokens_as_hashes.py b/indico/migrations/versions/20210222_1914_d354278c6d95_store_tokens_as_hashes.py new file mode 100644 index 00000000000..eb9fc8914b9 --- /dev/null +++ b/indico/migrations/versions/20210222_1914_d354278c6d95_store_tokens_as_hashes.py @@ -0,0 +1,69 @@ +"""Store tokens as hashes + +Revision ID: d354278c6d95 +Revises: c36abe1c23c7 +Create Date: 2021-02-22 19:14:03.752863 +""" + +import hashlib + +import sqlalchemy as sa +from alembic import context, op +from sqlalchemy.dialects import postgresql + +from indico.core.db.sqlalchemy.custom.utcdatetime import UTCDateTime +from indico.util.date_time import now_utc + + +# revision identifiers, used by Alembic. +revision = 'd354278c6d95' +down_revision = 'c36abe1c23c7' +branch_labels = None +depends_on = None + + +def sha256(token): + return hashlib.sha256(str(token).encode()).hexdigest() + + +def upgrade(): + if context.is_offline_mode(): + raise Exception('This upgrade is only possible in online mode') + conn = op.get_bind() + op.add_column('tokens', sa.Column('access_token_hash', sa.String(), nullable=True), schema='oauth') + op.add_column('tokens', sa.Column('created_dt', UTCDateTime(), nullable=True), schema='oauth') + conn.execute('UPDATE oauth.tokens SET created_dt = COALESCE(last_used_dt, %s)', (now_utc(),)) + op.alter_column('tokens', 'created_dt', nullable=False, schema='oauth') + op.create_index(None, 'tokens', ['access_token_hash'], unique=True, schema='oauth') + res = conn.execute('SELECT id, access_token FROM oauth.tokens') + for token_id, access_token in res: + conn.execute('UPDATE oauth.tokens SET access_token_hash = %s WHERE id = %s', (sha256(access_token), token_id)) + op.alter_column('tokens', 'access_token_hash', nullable=False, schema='oauth') + op.drop_column('tokens', 'access_token', schema='oauth') + op.drop_constraint('uq_tokens_app_user_link_id_scopes', 'tokens', schema='oauth') + + +def downgrade(): + op.execute(''' + DELETE FROM oauth.tokens + WHERE (app_user_link_id, scopes) IN ( + SELECT app_user_link_id, scopes FROM oauth.tokens GROUP BY app_user_link_id, scopes HAVING COUNT(*) > 1 + ); + ''') + op.create_unique_constraint(None, 'tokens', ['app_user_link_id', 'scopes'], schema='oauth') + op.add_column( + 'tokens', + sa.Column( + 'access_token', + postgresql.UUID(), + nullable=False, + server_default=sa.text(""" + uuid_in(overlay(overlay(md5(random()::text || ':' || clock_timestamp()::text) placing '4' from 13) + placing to_hex(floor(random()*(11-8+1) + 8)::int)::text from 17)::cstring)""") + ), + schema='oauth' + ) + op.alter_column('tokens', 'access_token', server_default=None, schema='oauth') + op.create_unique_constraint(None, 'tokens', ['access_token'], schema='oauth') + op.drop_column('tokens', 'access_token_hash', schema='oauth') + op.drop_column('tokens', 'created_dt', schema='oauth') diff --git a/indico/migrations/versions/20210224_1805_ecc7088914e7_use_cascading_fks_for_oauth.py b/indico/migrations/versions/20210224_1805_ecc7088914e7_use_cascading_fks_for_oauth.py new file mode 100644 index 00000000000..75b3d5001f8 --- /dev/null +++ b/indico/migrations/versions/20210224_1805_ecc7088914e7_use_cascading_fks_for_oauth.py @@ -0,0 +1,35 @@ +"""Use cascading FKs for oauth + +Revision ID: ecc7088914e7 +Revises: d354278c6d95 +Create Date: 2021-02-24 18:05:51.453397 +""" + +from alembic import op + + +# revision identifiers, used by Alembic. +revision = 'ecc7088914e7' +down_revision = 'd354278c6d95' +branch_labels = None +depends_on = None + + +def upgrade(): + op.drop_constraint('fk_application_user_links_application_id_applications', 'application_user_links', + schema='oauth') + op.drop_constraint('fk_application_user_links_user_id_users', 'application_user_links', schema='oauth') + op.create_foreign_key(None, 'application_user_links', 'applications', ['application_id'], ['id'], + source_schema='oauth', referent_schema='oauth', ondelete='CASCADE') + op.create_foreign_key(None, 'application_user_links', 'users', ['user_id'], ['id'], + source_schema='oauth', referent_schema='users', ondelete='CASCADE') + + +def downgrade(): + op.drop_constraint('fk_application_user_links_user_id_users', 'application_user_links', schema='oauth') + op.drop_constraint('fk_application_user_links_application_id_applications', 'application_user_links', + schema='oauth') + op.create_foreign_key(None, 'application_user_links', 'users', ['user_id'], ['id'], + source_schema='oauth', referent_schema='users') + op.create_foreign_key(None, 'application_user_links', 'applications', ['application_id'], ['id'], + source_schema='oauth', referent_schema='oauth') diff --git a/indico/migrations/versions/20210224_1808_26806768cd3f_remove_flower_oauth_app.py b/indico/migrations/versions/20210224_1808_26806768cd3f_remove_flower_oauth_app.py new file mode 100644 index 00000000000..d2b851444e6 --- /dev/null +++ b/indico/migrations/versions/20210224_1808_26806768cd3f_remove_flower_oauth_app.py @@ -0,0 +1,37 @@ +"""Remove Flower oauth app + +Revision ID: 26806768cd3f +Revises: ecc7088914e7 +Create Date: 2021-02-24 18:08:05.634719 +""" + +from uuid import uuid4 + +from alembic import op + + +# revision identifiers, used by Alembic. +revision = '26806768cd3f' +down_revision = 'ecc7088914e7' +branch_labels = None +depends_on = None + + +def upgrade(): + op.execute('DELETE FROM oauth.applications WHERE system_app_type = 2') + op.drop_constraint('ck_applications_valid_enum_system_app_type', 'applications', schema='oauth') + op.create_check_constraint('valid_enum_system_app_type', 'applications', '(system_app_type = ANY (ARRAY[0, 1]))', + schema='oauth') + + +def downgrade(): + op.drop_constraint('ck_applications_valid_enum_system_app_type', 'applications', schema='oauth') + op.create_check_constraint('valid_enum_system_app_type', 'applications', '(system_app_type = ANY (ARRAY[0, 1, 2]))', + schema='oauth') + op.execute(f''' + INSERT INTO oauth.applications + (name, description, client_id, client_secret, allowed_scopes, redirect_uris, is_enabled, is_trusted, + system_app_type, allow_pkce_flow) + VALUES + ('Flower', '', '{uuid4()}', '{uuid4()}', '{{read:user}}', '{{}}', true, true, 2, false); + ''') diff --git a/indico/modules/__init__.py b/indico/modules/__init__.py index dc6c06a6f76..8115d0194c9 100644 --- a/indico/modules/__init__.py +++ b/indico/modules/__init__.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/modules/admin/__init__.py b/indico/modules/admin/__init__.py index feaff6f0714..1d67ec4660c 100644 --- a/indico/modules/admin/__init__.py +++ b/indico/modules/admin/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals diff --git a/indico/modules/admin/controllers/base.py b/indico/modules/admin/controllers/base.py index 2b24c1a74ec..eb29de904af 100644 --- a/indico/modules/admin/controllers/base.py +++ b/indico/modules/admin/controllers/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from werkzeug.exceptions import Forbidden @@ -15,7 +13,7 @@ class RHAdminBase(RHProtected): - """Base class for all admin-only RHs""" + """Base class for all admin-only RHs.""" DENY_FRAMES = True diff --git a/indico/modules/admin/views.py b/indico/modules/admin/views.py index e9ecbb6c8ab..4aa29681b95 100644 --- a/indico/modules/admin/views.py +++ b/indico/modules/admin/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.util.i18n import _ from indico.web.breadcrumbs import render_breadcrumbs from indico.web.flask.util import url_for diff --git a/indico/modules/announcement/__init__.py b/indico/modules/announcement/__init__.py index 5195de41026..e8c7d44e11c 100644 --- a/indico/modules/announcement/__init__.py +++ b/indico/modules/announcement/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals diff --git a/indico/modules/announcement/blueprint.py b/indico/modules/announcement/blueprint.py index a897e150492..3d176dcbb50 100644 --- a/indico/modules/announcement/blueprint.py +++ b/indico/modules/announcement/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.announcement.controllers import RHAnnouncement from indico.web.flask.wrappers import IndicoBlueprint diff --git a/indico/modules/announcement/controllers.py b/indico/modules/announcement/controllers.py index 9795a1998d9..7e6b3a99506 100644 --- a/indico/modules/announcement/controllers.py +++ b/indico/modules/announcement/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, redirect from indico.modules.admin import RHAdminBase diff --git a/indico/modules/announcement/forms.py b/indico/modules/announcement/forms.py index b89d3ee2973..ec5a0c1ab99 100644 --- a/indico/modules/announcement/forms.py +++ b/indico/modules/announcement/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms.fields import BooleanField, TextAreaField from wtforms.validators import DataRequired diff --git a/indico/modules/announcement/views.py b/indico/modules/announcement/views.py index 58e518636b6..cff35ba7c4c 100644 --- a/indico/modules/announcement/views.py +++ b/indico/modules/announcement/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.admin.views import WPAdmin diff --git a/indico/modules/api/__init__.py b/indico/modules/api/__init__.py index 0269d7119d7..b308c5d39e8 100644 --- a/indico/modules/api/__init__.py +++ b/indico/modules/api/__init__.py @@ -1,20 +1,18 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals from indico.core.db import db from indico.core.settings import SettingsProxy from indico.modules.api.models.keys import APIKey +from indico.util.enum import IndicoEnum from indico.util.i18n import _ -from indico.util.struct.enum import IndicoEnum from indico.web.flask.util import url_for from indico.web.menu import SideMenuItem @@ -44,7 +42,7 @@ def _merge_users(target, source, **kwargs): ak_user = target.api_key ak_merged = source.api_key # Move all inactive keys to the new user - APIKey.find(user_id=source.id, is_active=False).update({'user_id': target.id}) + APIKey.query.filter_by(user_id=source.id, is_active=False).update({'user_id': target.id}) if ak_merged and not ak_user: ak_merged.user = target elif ak_user and ak_merged: diff --git a/indico/modules/api/blueprint.py b/indico/modules/api/blueprint.py index 13d04ff6284..b6b8c711764 100644 --- a/indico/modules/api/blueprint.py +++ b/indico/modules/api/blueprint.py @@ -1,32 +1,25 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request -from indico.legacy.services.interface.rpc.json import process as jsonrpc_handler -from indico.modules.api.controllers import (RHAPIAdminKeys, RHAPIAdminSettings, RHAPIBlockKey, RHAPIBuildURLs, - RHAPICreateKey, RHAPIDeleteKey, RHAPITogglePersistent, RHAPIUserProfile) +from indico.modules.api.controllers import (RHAPIAdminKeys, RHAPIAdminSettings, RHAPIBlockKey, RHAPICreateKey, + RHAPIDeleteKey, RHAPITogglePersistent, RHAPIUserProfile) from indico.web.flask.wrappers import IndicoBlueprint from indico.web.http_api.handlers import handler as api_handler _bp = IndicoBlueprint('api', __name__, template_folder='templates', virtual_template_folder='api') -# Legacy JSON-RPC API -_bp.add_url_rule('/services/json-rpc', view_func=jsonrpc_handler, endpoint='jsonrpc', methods=('POST',)) - # HTTP API _bp.add_url_rule('/export/', view_func=api_handler, endpoint='httpapi', defaults={'prefix': 'export'}) _bp.add_url_rule('/api/', view_func=api_handler, endpoint='httpapi', defaults={'prefix': 'api'}, methods=('POST',)) _bp.add_url_rule('/', endpoint='httpapi', build_only=True) -_bp.add_url_rule('/api/build-urls', 'build_urls', RHAPIBuildURLs, methods=('POST',)) # Administration _bp.add_url_rule('/admin/api/', 'admin_settings', RHAPIAdminSettings, methods=('GET', 'POST')) diff --git a/indico/modules/api/controllers.py b/indico/modules/api/controllers.py index 142f862691d..57eaeb167a9 100644 --- a/indico/modules/api/controllers.py +++ b/indico/modules/api/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, redirect, request, session from werkzeug.exceptions import BadRequest, Forbidden @@ -16,21 +14,14 @@ from indico.modules.api.forms import AdminSettingsForm from indico.modules.api.models.keys import APIKey from indico.modules.api.views import WPAPIAdmin, WPAPIUserProfile -from indico.modules.categories.models.categories import Category -from indico.modules.events.contributions.models.contributions import Contribution -from indico.modules.events.models.events import Event -from indico.modules.events.sessions.models.sessions import Session from indico.modules.users.controllers import RHUserBase from indico.util.i18n import _ from indico.web.flask.util import redirect_or_jsonify, url_for from indico.web.forms.base import FormDefaults -from indico.web.http_api.util import generate_public_auth_request -from indico.web.rh import RH -from indico.web.util import jsonify_data class RHAPIAdminSettings(RHAdminBase): - """API settings (admin)""" + """API settings (admin).""" def _process(self): form = AdminSettingsForm(obj=FormDefaults(**api_settings.get_all())) @@ -38,26 +29,26 @@ def _process(self): api_settings.set_multi(form.data) flash(_('Settings saved'), 'success') return redirect(url_for('.admin_settings')) - count = APIKey.find(is_active=True).count() + count = APIKey.query.filter_by(is_active=True).count() return WPAPIAdmin.render_template('admin_settings.html', form=form, count=count) class RHAPIAdminKeys(RHAdminBase): - """API key list (admin)""" + """API key list (admin).""" def _process(self): - keys = sorted(APIKey.find_all(is_active=True), key=lambda ak: (ak.use_count == 0, ak.user.full_name)) + keys = sorted(APIKey.query.filter_by(is_active=True), key=lambda ak: (ak.use_count == 0, ak.user.full_name)) return WPAPIAdmin.render_template('admin_keys.html', keys=keys) class RHUserAPIBase(RHUserBase): - """Base class for user API management""" + """Base class for user API management.""" allow_system_user = True class RHAPIUserProfile(RHUserAPIBase): - """API key details (user)""" + """API key details (user).""" def _process(self): key = self.user.api_key @@ -72,7 +63,7 @@ def _process(self): class RHAPICreateKey(RHUserAPIBase): - """API key creation""" + """API key creation.""" def _process(self): quiet = request.form.get('quiet') == '1' @@ -105,7 +96,7 @@ def _process(self): class RHAPIDeleteKey(RHUserAPIBase): - """API key deletion""" + """API key deletion.""" def _process(self): key = self.user.api_key @@ -115,7 +106,7 @@ def _process(self): class RHAPITogglePersistent(RHUserAPIBase): - """API key - persistent signatures on/off""" + """API key - persistent signatures on/off.""" def _process(self): quiet = request.form.get('quiet') == '1' @@ -130,7 +121,7 @@ def _process(self): class RHAPIBlockKey(RHUserAPIBase): - """API key blocking/unblocking""" + """API key blocking/unblocking.""" def _check_access(self): RHUserAPIBase._check_access(self) @@ -145,41 +136,3 @@ def _process(self): else: flash(_('The API key has been unblocked.'), 'success') return redirect(url_for('api.user_profile')) - - -class RHAPIBuildURLs(RH): - def _process_args(self): - data = request.json - self.object = None - if 'categId' in data: - self.object = Category.get_or_404(data['categId']) - elif 'contribId' in data: - self.object = Contribution.get_or_404(data['contribId']) - elif 'sessionId' in data: - self.object = Session.get_or_404(data['sessionId']) - elif 'confId' in data: - self.object = Event.get_or_404(data['confId']) - - if self.object is None: - raise BadRequest - - def _process(self): - urls = {} - api_key = session.user.api_key if session.user else None - url_format = '/export/event/{0}/{1}/{2}.ics' - if isinstance(self.object, Contribution): - event = self.object.event - urls = generate_public_auth_request(api_key, url_format.format(event.id, 'contribution', self.object.id)) - elif isinstance(self.object, Session): - event = self.object.event - urls = generate_public_auth_request(api_key, url_format.format(event.id, 'session', self.object.id)) - elif isinstance(self.object, Category): - urls = generate_public_auth_request(api_key, '/export/categ/{0}.ics'.format(self.object.id), - {'from': '-31d'}) - elif isinstance(self.object, Event): - urls = generate_public_auth_request(api_key, '/export/event/{0}.ics'.format(self.object.id)) - event_urls = generate_public_auth_request(api_key, '/export/event/{0}.ics'.format(self.object.id), - {'detail': 'contribution'}) - urls['publicRequestDetailedURL'] = event_urls['publicRequestURL'] - urls['authRequestDetailedURL'] = event_urls['authRequestURL'] - return jsonify_data(flash=False, urls=urls) diff --git a/indico/modules/api/forms.py b/indico/modules/api/forms.py index 1e0cd8a8de4..b16284c0431 100644 --- a/indico/modules/api/forms.py +++ b/indico/modules/api/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms.fields.core import BooleanField from wtforms.fields.html5 import IntegerField from wtforms.validators import NumberRange diff --git a/indico/modules/api/models/keys.py b/indico/modules/api/models/keys.py index 42e955eb4d0..779eb13bc72 100644 --- a/indico/modules/api/models/keys.py +++ b/indico/modules/api/models/keys.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from uuid import uuid4 from sqlalchemy.dialects.postgresql import INET, UUID @@ -14,11 +12,10 @@ from indico.core.db import db from indico.core.db.sqlalchemy import UTCDateTime from indico.util.date_time import now_utc -from indico.util.string import return_ascii class APIKey(db.Model): - """API keys for users""" + """API keys for users.""" __tablename__ = 'api_keys' __table_args__ = (db.Index(None, 'user_id', unique=True, postgresql_where=db.text('is_active')), {'schema': 'users'}) @@ -33,13 +30,13 @@ class APIKey(db.Model): UUID, nullable=False, unique=True, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) #: secret key used for signed requests secret = db.Column( UUID, nullable=False, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) #: ID of the user associated with the key user_id = db.Column( @@ -105,12 +102,11 @@ class APIKey(db.Model): lazy=False ) - @return_ascii def __repr__(self): return ''.format(self.token, self.user_id, self.last_used_dt or 'never') def register_used(self, ip, uri, authenticated): - """Updates the last used information""" + """Update the last used information.""" self.last_used_dt = now_utc() self.last_used_ip = ip self.last_used_uri = uri diff --git a/indico/modules/api/templates/_messages.html b/indico/modules/api/templates/_messages.html index daa8a1e81de..30cc1f9d2dc 100644 --- a/indico/modules/api/templates/_messages.html +++ b/indico/modules/api/templates/_messages.html @@ -15,27 +15,3 @@ invalidate them, you have to create a new API key! {%- endtrans -%} {%- endmacro %} - -{% macro get_ical_api_key_msg() -%} - {%- trans -%} - In order to enable an iCal export link, your account needs to have an API key created. - This key enables other applications to access data from within Indico even when you are - neither using nor logged into the Indico system yourself with the link provided. - Once created, you can manage your key at any time by going to 'My Profile' and looking - under the tab entitled 'HTTP API'. Further information about HTTP API keys can be found - in the Indico documentation. - {%- endtrans -%} -{%- endmacro %} - - -{% macro get_ical_persistent_msg() -%} - {%- trans -%} - Additionally to having an API key associated with your account, exporting private event - information requires the usage of a persistent signature. This enables API URLs which do - not expire after a few minutes so while the setting is active, anyone in possession of the - link provided can access the information. Due to this, it is extremely important that you keep - these links private and for your use only. If you think someone else may have acquired access - to a link using this key in the future, you must immediately create a new key pair on the - 'My Profile' page under the 'HTTP API' and update the iCalendar links afterwards. - {%- endtrans -%} -{%- endmacro %} diff --git a/indico/modules/api/templates/user_profile.html b/indico/modules/api/templates/user_profile.html index 8a66d4a30fe..c894e7fea76 100644 --- a/indico/modules/api/templates/user_profile.html +++ b/indico/modules/api/templates/user_profile.html @@ -185,7 +185,7 @@ used {{ n }} times {%- endtrans -%} {%- if old_key.last_used_dt -%} - , {% trans date=old_key.last_used_dt|format_datetime('short')|ensure_unicode -%} + , {% trans date=old_key.last_used_dt|format_datetime('short') -%} last used on {{ date }} {%- endtrans %} {%- endif -%} diff --git a/indico/modules/api/views.py b/indico/modules/api/views.py index c21b9dea8cd..fb6826d61fc 100644 --- a/indico/modules/api/views.py +++ b/indico/modules/api/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.admin.views import WPAdmin from indico.modules.users.views import WPUser diff --git a/indico/modules/attachments/__init__.py b/indico/modules/attachments/__init__.py index 4b1ab87b3ba..4aac2112340 100644 --- a/indico/modules/attachments/__init__.py +++ b/indico/modules/attachments/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals @@ -29,9 +27,9 @@ @signals.users.merged.connect def _merge_users(target, source, **kwargs): from indico.modules.attachments.models.attachments import Attachment, AttachmentFile - from indico.modules.attachments.models.principals import AttachmentPrincipal, AttachmentFolderPrincipal - Attachment.find(user_id=source.id).update({Attachment.user_id: target.id}) - AttachmentFile.find(user_id=source.id).update({AttachmentFile.user_id: target.id}) + from indico.modules.attachments.models.principals import AttachmentFolderPrincipal, AttachmentPrincipal + Attachment.query.filter_by(user_id=source.id).update({Attachment.user_id: target.id}) + AttachmentFile.query.filter_by(user_id=source.id).update({AttachmentFile.user_id: target.id}) AttachmentPrincipal.merge_users(target, source, 'attachment') AttachmentFolderPrincipal.merge_users(target, source, 'folder') @@ -59,3 +57,8 @@ def _extend_category_management_menu(sender, category, **kwargs): def _get_attachment_cloner(sender, **kwargs): from indico.modules.attachments.clone import AttachmentCloner return AttachmentCloner + + +@signals.import_tasks.connect +def _import_tasks(sender, **kwargs): + import indico.modules.attachments.tasks # noqa: F401 diff --git a/indico/modules/attachments/api/hooks.py b/indico/modules/attachments/api/hooks.py index 2fe4d1d6eba..4f8454d4799 100644 --- a/indico/modules/attachments/api/hooks.py +++ b/indico/modules/attachments/api/hooks.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.attachments.api.util import build_folders_api_data from indico.modules.events import Event from indico.modules.events.contributions.models.contributions import Contribution @@ -26,7 +24,7 @@ class AttachmentsExportHook(HTTPAPIHook): VALID_FORMATS = ('json', 'jsonp', 'xml') def _getParams(self): - super(AttachmentsExportHook, self)._getParams() + super()._getParams() event = self._obj = Event.get(self._pathParams['event_id'], is_deleted=False) if event is None: raise HTTPAPIError('No such event', 404) diff --git a/indico/modules/attachments/api/util.py b/indico/modules/attachments/api/util.py index 676b4122568..f8601e227c5 100644 --- a/indico/modules/attachments/api/util.py +++ b/indico/modules/attachments/api/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import defaultdict from flask import g @@ -32,7 +30,7 @@ def build_material_legacy_api_data(linked_object): for folder in query: cache[folder.object].append(folder) - return filter(None, map(_build_folder_legacy_api_data, cache.get(linked_object, []))) + return [_f for _f in map(_build_folder_legacy_api_data, cache.get(linked_object, [])) if _f] def _build_folder_legacy_api_data(folder): @@ -83,7 +81,7 @@ def build_folders_api_data(linked_object): folders = get_attached_folders(linked_object, preload_event=True) if not folders: return [] - return filter(None, map(_build_folder_api_data, folders)) + return [_f for _f in map(_build_folder_api_data, folders) if _f] def _build_folder_api_data(folder): diff --git a/indico/modules/attachments/blueprint.py b/indico/modules/attachments/blueprint.py index 1f005635c75..fc04e1c09a9 100644 --- a/indico/modules/attachments/blueprint.py +++ b/indico/modules/attachments/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import itertools from indico.modules.attachments.controllers.compat import (RHCompatAttachmentNew, compat_attachment, compat_folder, @@ -15,6 +13,7 @@ from indico.modules.attachments.controllers.display.event import (RHDownloadEventAttachment, RHListEventAttachmentFolder, RHPackageEventAttachmentsDisplay) +from indico.modules.attachments.controllers.event_package import RHPackageEventAttachmentsStatus from indico.modules.attachments.controllers.management.category import (RHAddCategoryAttachmentFiles, RHAddCategoryAttachmentLink, RHCreateCategoryFolder, @@ -50,13 +49,13 @@ def view_func(**kwargs): # Management -items = itertools.chain(event_management_object_url_prefixes.iteritems(), [('category', ['/manage'])]) +items = itertools.chain(event_management_object_url_prefixes.items(), [('category', ['/manage'])]) for object_type, prefixes in items: for prefix in prefixes: if object_type == 'category': prefix = '/category/' + prefix else: - prefix = '/event/' + prefix + prefix = '/event/' + prefix _bp.add_url_rule(prefix + '/attachments/', 'management', _dispatch(RHManageEventAttachments, RHManageCategoryAttachments), defaults={'object_type': object_type}) @@ -86,13 +85,13 @@ def view_func(**kwargs): methods=('DELETE',), defaults={'object_type': object_type}) # Display/download -items = itertools.chain(event_object_url_prefixes.iteritems(), [('category', [''])]) +items = itertools.chain(event_object_url_prefixes.items(), [('category', [''])]) for object_type, prefixes in items: for prefix in prefixes: if object_type == 'category': prefix = '/category/' + prefix else: - prefix = '/event/' + prefix + prefix = '/event/' + prefix _bp.add_url_rule(prefix + '/attachments///', 'download', _dispatch(RHDownloadEventAttachment, RHDownloadCategoryAttachment), defaults={'object_type': object_type}) @@ -105,14 +104,16 @@ def view_func(**kwargs): # Package -_bp.add_url_rule('/event//attachments/package', 'package', +_bp.add_url_rule('/event//attachments/package', 'package', RHPackageEventAttachmentsDisplay, methods=('GET', 'POST')) -_bp.add_url_rule('/event//manage/attachments/package', 'package_management', +_bp.add_url_rule('/event//manage/attachments/package', 'package_management', RHPackageEventAttachmentsManagement, methods=('GET', 'POST')) +_bp.add_url_rule('/event//attachments/package/status/', 'package_status', + RHPackageEventAttachmentsStatus) # Legacy redirects for the old URLs -_compat_bp = IndicoBlueprint('compat_attachments', __name__, url_prefix='/event/') +_compat_bp = IndicoBlueprint('compat_attachments', __name__, url_prefix='/event/') compat_folder_rules = [ '/material//', '/session//contribution//material//', @@ -133,17 +134,17 @@ def view_func(**kwargs): '/contribution///material//.' ] old_obj_prefix_rules = { - 'session': ['!/event//session/'], - 'contribution': ['!/event//session//contribution/', - '!/event//contribution/'], - 'subcontribution': ['!/event//session//contribution//', - '!/event//contribution//'] + 'session': ['!/event//session/'], + 'contribution': ['!/event//session//contribution/', + '!/event//contribution/'], + 'subcontribution': ['!/event//session//contribution//', + '!/event//contribution//'] } for rule in compat_folder_rules: _compat_bp.add_url_rule(rule, 'folder', compat_folder) for rule in compat_attachment_rules: _compat_bp.add_url_rule(rule, 'attachment', compat_attachment) -for object_type, prefixes in old_obj_prefix_rules.iteritems(): +for object_type, prefixes in old_obj_prefix_rules.items(): for prefix in prefixes: # we rely on url normalization to redirect to the proper URL for the object _compat_bp.add_url_rule(prefix + '/attachments///', diff --git a/indico/modules/attachments/client/js/index.js b/indico/modules/attachments/client/js/index.js index 44286fbdf28..702639b8175 100644 --- a/indico/modules/attachments/client/js/index.js +++ b/indico/modules/attachments/client/js/index.js @@ -1,316 +1,9 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -(function(global) { - 'use strict'; - - var HISTORY_API_SUPPORTED = !!history.pushState; - - function toggleFolder(evt) { - if ($(evt.target).closest('.actions').length) { - // ignore if it comes from inside the action panel - return; - } - $(this) - .toggleClass('collapsed') - .next('.sub-tree') - .find('td > div') - .slideToggle(150); - } - - $(document).ready(function() { - $('.attachments > .i-dropdown') - .parent() - .dropdown(); - if (!$('html').data('static-site')) { - setupAttachmentPreview(); - } - - $(document).on('click', '[data-attachment-editor]', function(evt) { - evt.preventDefault(); - var $this = $(this); - if (this.disabled || $this.hasClass('disabled')) { - return; - } - var locator = $(this).data('locator'); - var title = $(this).data('title'); - var reloadOnChange = $this.data('reload-on-change') !== undefined; - openAttachmentManager(locator, title, reloadOnChange, $this); - }); - }); - - global.setupAttachmentPreview = function setupAttachmentPreview() { - var attachment = $('.js-preview-dialog'); - var pageURL = location.href.replace(/#.*$/, ''); - - // Previewer not supported on mobile browsers - if ($.mobileBrowser) { - return; - } - - $(window) - .on('hashchange', function(e, initial) { - if (location.hash.indexOf('#preview:') !== 0) { - $('.attachment-preview-dialog').trigger('ajaxDialog:close', [true]); - } else { - if (initial && HISTORY_API_SUPPORTED) { - var hash = location.hash; - // start with a clean state, i.e. [..., page, page+preview] - history.replaceState({}, document.title, pageURL); - history.pushState({}, document.title, location.href + hash); - } - var id = location.hash.split('#preview:')[1]; - previewAttachment(id); - } - }) - .triggerHandler('hashchange', [true]); - - attachment.on('click', function(e) { - if (e.which != 1 || e.shiftKey || e.metaKey || e.ctrlKey || e.altKey) { - // ignore middle clicks and modifier-clicks - people should be able to open - // an attachment in a new tab/window skipping the previewer, even if they use - // a weird mouse with less than three buttons. - return; - } - e.preventDefault(); - location.hash = '#preview:{0}'.format($(this).data('attachmentId')); - }); - - function clearHash() { - if (HISTORY_API_SUPPORTED) { - // if we have the history api, we can assume that the previous state is the same page without - // a preview hash (since we ensure this in the initial/fake hashchange event on page load) - history.back(); - } else { - // old browsers with no pushState support: the # wil stay which is a bit ugly, - // but let's not break history (we WILL "spam" history though, but that's what you - // get when using ancient browsers) - location.hash = ''; - } - } - - function previewAttachment(id) { - var attachment = $('.attachment[data-previewable][data-attachment-id="{0}"]'.format(id)); - if (!attachment.length) { - clearHash(); - return; - } - ajaxDialog({ - url: build_url(attachment.attr('href'), {preview: '1'}), - title: attachment.data('title'), - dialogClasses: 'attachment-preview-dialog', - onClose: function(data) { - $('body').off('keydown.attachmentPreview'); - $('html, body').removeClass('prevent-scrolling'); - if (!data) { - // get rid of the hash if we closed the dialog manually (i.e. not using the back button, - // in which case the hash should already be gone) - clearHash(); - } - }, - onOpen: function(popup) { - var dialog = popup.canvas.closest('.ui-dialog'); - dialog.prev('.ui-widget-overlay').addClass('attachment-preview-overlay'); - popup.canvas - .find('.attachment-preview-content-wrapper, .js-close-preview') - .on('click', function() { - popup.canvas.trigger('ajaxDialog:close'); - }); - popup.canvas.find('.attachment-download').on('click', function() { - var $this = $(this); - var href = $this.attr('href'); - $this.attr('href', build_url(href, {from_preview: '1', download: '1'})); - _.defer(function() { - $this.attr('href', href); - }); - }); - popup.canvas - .find('.attachment-preview-content, .attachment-preview-top-bar') - .on('click', function(e) { - e.stopPropagation(); - }); - $('body') - .add(dialog) - .on('keydown.attachmentPreview', function(e) { - if (e.which === $.ui.keyCode.ESCAPE) { - popup.canvas.trigger('ajaxDialog:close'); - } - }); - $('html, body').addClass('prevent-scrolling'); - // for some reason the dialog is hidden when its position - // gets updated so we explicitly show it. - _.defer(function() { - dialog.show(); - }); - }, - onLoadError: function(xhr) { - var hash = location.hash; - clearHash(); - if (xhr.status == 404) { - alertPopup($T.gettext('This file no longer exists. Please reload the page.')); - return false; - } else if (xhr.status != 403) { - return; - } - if (Indico.User && Indico.User.id !== undefined) { - alertPopup($T('You are not authorized to access this file.'), $T('Access Denied')); - } else { - var msg = $T('This file is protected. You will be redirected to the login page.'); - confirmPrompt(msg, $T('Access Denied')).then(function() { - location.href = build_url(Indico.Urls.Login, {next: location.href + hash}); - }); - } - return false; - }, - }); - } - }; - - global.setupAttachmentTreeView = function setupAttachmentTreeView() { - $('.attachments-box').on('click', '.tree .expandable', toggleFolder); - }; - - global.setupAttachmentEditor = function setupAttachmentEditor() { - var editor = $('.attachment-editor'); - - function flagChanged() { - editor.trigger('ajaxDialog:setData', [true]); - } - - editor - .on('click', '.tree .expandable', toggleFolder) - .on('click', '.js-dialog-action', function(e) { - e.preventDefault(); - var $this = $(this); - ajaxDialog({ - trigger: this, - url: $this.data('href'), - title: $this.data('title'), - hidePageHeader: true, - onClose: function(data) { - if (data) { - $('#attachments-container').html(data.attachment_list); - flagChanged(); - } - }, - }); - }) - .on('indico:confirmed', '.js-delete', function(e) { - e.preventDefault(); - - var $this = $(this); - $.ajax({ - url: $this.data('href'), - method: $this.data('method'), - complete: IndicoUI.Dialogs.Util.progress(), - error: handleAjaxError, - success: function(data) { - $('#attachments-container').html(data.attachment_list); - handleFlashes(data, true, editor); - flagChanged(); - }, - }); - }); - }; - - global.openAttachmentManager = function openAttachmentManager( - itemLocator, - title, - reloadOnChange, - trigger - ) { - reloadOnChange = reloadOnChange === undefined ? true : reloadOnChange; - ajaxDialog({ - trigger: trigger, - url: build_url(Indico.Urls.AttachmentManager, itemLocator), - title: title || $T.gettext('Manage material'), - confirmCloseUnsaved: false, - hidePageHeader: true, - onClose: function(callbackData, customData) { - if (customData && reloadOnChange) { - location.reload(); - } else if (customData && trigger) { - trigger.trigger('attachments:updated'); - } - }, - }); - }; - - global.reloadManagementAttachmentInfoColumn = function reloadManagementAttachmentInfoColumn( - itemLocator, - column - ) { - $.ajax({ - url: build_url(Indico.Urls.ManagementAttachmentInfoColumn, itemLocator), - method: 'GET', - error: handleAjaxError, - success: function(data) { - column.replaceWith(data.html); - }, - }); - }; - - global.messageIfFolderProtected = function messageIfFolderProtected( - protectionField, - folderField, - protectionInfo, - selfProtection, - inheritedProtection, - folderProtection - ) { - folderField.on('change', function() { - var selectedFolder = $(this); - if (protectionInfo[selectedFolder.val()] && !protectionField.prop('checked')) { - selfProtection.hide(); - inheritedProtection.hide(); - folderProtection - .find('.folder-name') - .html(selectedFolder.children('option:selected').text()); - folderProtection.show(); - } else { - folderProtection.hide(); - selfProtection.toggle(protectionField.prop('checked')); - inheritedProtection.toggle(!protectionField.prop('checked')); - } - }); - _.defer(function() { - folderField.triggerHandler('change'); - }); - }; - - global.setupAttachmentTooltipButtons = function setupAttachmentTooltipButtons() { - $('.attachments-tooltip-button').each(function() { - var button = $(this); - button.qtip({ - content: { - text: button.next('.material_list'), - }, - show: { - event: 'click', - }, - hide: { - event: 'unfocus', - }, - position: { - my: 'top right', - at: 'bottom left', - }, - events: { - show: function() { - button.addClass('open'); - }, - hide: function() { - button.removeClass('open'); - }, - }, - style: { - classes: 'material_tip', - }, - }); - }); - }; -})(window); +import './legacy'; +import './package'; diff --git a/indico/modules/attachments/client/js/legacy.js b/indico/modules/attachments/client/js/legacy.js new file mode 100644 index 00000000000..a96a3cb3325 --- /dev/null +++ b/indico/modules/attachments/client/js/legacy.js @@ -0,0 +1,316 @@ +// This file is part of Indico. +// Copyright (C) 2002 - 2021 CERN +// +// Indico is free software; you can redistribute it and/or +// modify it under the terms of the MIT License; see the +// LICENSE file for more details. + +(function(global) { + 'use strict'; + + var HISTORY_API_SUPPORTED = !!history.pushState; + + function toggleFolder(evt) { + if ($(evt.target).closest('.actions').length) { + // ignore if it comes from inside the action panel + return; + } + $(this) + .toggleClass('collapsed') + .next('.sub-tree') + .find('td > div') + .slideToggle(150); + } + + $(document).ready(function() { + $('.attachments > .i-dropdown') + .parent() + .dropdown(); + if (!$('html').data('static-site')) { + setupAttachmentPreview(); + } + + $(document).on('click', '[data-attachment-editor]', function(evt) { + evt.preventDefault(); + var $this = $(this); + if (this.disabled || $this.hasClass('disabled')) { + return; + } + var locator = $(this).data('locator'); + var title = $(this).data('title'); + var reloadOnChange = $this.data('reload-on-change') !== undefined; + openAttachmentManager(locator, title, reloadOnChange, $this); + }); + }); + + global.setupAttachmentPreview = function setupAttachmentPreview() { + var attachment = $('.js-preview-dialog'); + var pageURL = location.href.replace(/#.*$/, ''); + + // Previewer not supported on mobile browsers + if ($.mobileBrowser) { + return; + } + + $(window) + .on('hashchange', function(e, initial) { + if (location.hash.indexOf('#preview:') !== 0) { + $('.attachment-preview-dialog').trigger('ajaxDialog:close', [true]); + } else { + if (initial && HISTORY_API_SUPPORTED) { + var hash = location.hash; + // start with a clean state, i.e. [..., page, page+preview] + history.replaceState({}, document.title, pageURL); + history.pushState({}, document.title, location.href + hash); + } + var id = location.hash.split('#preview:')[1]; + previewAttachment(id); + } + }) + .triggerHandler('hashchange', [true]); + + attachment.on('click', function(e) { + if (e.which != 1 || e.shiftKey || e.metaKey || e.ctrlKey || e.altKey) { + // ignore middle clicks and modifier-clicks - people should be able to open + // an attachment in a new tab/window skipping the previewer, even if they use + // a weird mouse with less than three buttons. + return; + } + e.preventDefault(); + location.hash = '#preview:{0}'.format($(this).data('attachmentId')); + }); + + function clearHash() { + if (HISTORY_API_SUPPORTED) { + // if we have the history api, we can assume that the previous state is the same page without + // a preview hash (since we ensure this in the initial/fake hashchange event on page load) + history.back(); + } else { + // old browsers with no pushState support: the # wil stay which is a bit ugly, + // but let's not break history (we WILL "spam" history though, but that's what you + // get when using ancient browsers) + location.hash = ''; + } + } + + function previewAttachment(id) { + var attachment = $('.attachment[data-previewable][data-attachment-id="{0}"]'.format(id)); + if (!attachment.length) { + clearHash(); + return; + } + ajaxDialog({ + url: build_url(attachment.attr('href'), {preview: '1'}), + title: attachment.data('title'), + dialogClasses: 'attachment-preview-dialog', + onClose: function(data) { + $('body').off('keydown.attachmentPreview'); + $('html, body').removeClass('prevent-scrolling'); + if (!data) { + // get rid of the hash if we closed the dialog manually (i.e. not using the back button, + // in which case the hash should already be gone) + clearHash(); + } + }, + onOpen: function(popup) { + var dialog = popup.canvas.closest('.ui-dialog'); + dialog.prev('.ui-widget-overlay').addClass('attachment-preview-overlay'); + popup.canvas + .find('.attachment-preview-content-wrapper, .js-close-preview') + .on('click', function() { + popup.canvas.trigger('ajaxDialog:close'); + }); + popup.canvas.find('.attachment-download').on('click', function() { + var $this = $(this); + var href = $this.attr('href'); + $this.attr('href', build_url(href, {from_preview: '1', download: '1'})); + _.defer(function() { + $this.attr('href', href); + }); + }); + popup.canvas + .find('.attachment-preview-content, .attachment-preview-top-bar') + .on('click', function(e) { + e.stopPropagation(); + }); + $('body') + .add(dialog) + .on('keydown.attachmentPreview', function(e) { + if (e.which === $.ui.keyCode.ESCAPE) { + popup.canvas.trigger('ajaxDialog:close'); + } + }); + $('html, body').addClass('prevent-scrolling'); + // for some reason the dialog is hidden when its position + // gets updated so we explicitly show it. + _.defer(function() { + dialog.show(); + }); + }, + onLoadError: function(xhr) { + var hash = location.hash; + clearHash(); + if (xhr.status == 404) { + alertPopup($T.gettext('This file no longer exists. Please reload the page.')); + return false; + } else if (xhr.status != 403) { + return; + } + if (Indico.User && Indico.User.id !== undefined) { + alertPopup($T('You are not authorized to access this file.'), $T('Access Denied')); + } else { + var msg = $T('This file is protected. You will be redirected to the login page.'); + confirmPrompt(msg, $T('Access Denied')).then(function() { + location.href = build_url(Indico.Urls.Login, {next: location.href + hash}); + }); + } + return false; + }, + }); + } + }; + + global.setupAttachmentTreeView = function setupAttachmentTreeView() { + $('.attachments-box').on('click', '.tree .expandable', toggleFolder); + }; + + global.setupAttachmentEditor = function setupAttachmentEditor() { + var editor = $('.attachment-editor'); + + function flagChanged() { + editor.trigger('ajaxDialog:setData', [true]); + } + + editor + .on('click', '.tree .expandable', toggleFolder) + .on('click', '.js-dialog-action', function(e) { + e.preventDefault(); + var $this = $(this); + ajaxDialog({ + trigger: this, + url: $this.data('href'), + title: $this.data('title'), + hidePageHeader: true, + onClose: function(data) { + if (data) { + $('#attachments-container').html(data.attachment_list); + flagChanged(); + } + }, + }); + }) + .on('indico:confirmed', '.js-delete', function(e) { + e.preventDefault(); + + var $this = $(this); + $.ajax({ + url: $this.data('href'), + method: $this.data('method'), + complete: IndicoUI.Dialogs.Util.progress(), + error: handleAjaxError, + success: function(data) { + $('#attachments-container').html(data.attachment_list); + handleFlashes(data, true, editor); + flagChanged(); + }, + }); + }); + }; + + global.openAttachmentManager = function openAttachmentManager( + itemLocator, + title, + reloadOnChange, + trigger + ) { + reloadOnChange = reloadOnChange === undefined ? true : reloadOnChange; + ajaxDialog({ + trigger: trigger, + url: build_url(Indico.Urls.AttachmentManager, itemLocator), + title: title || $T.gettext('Manage material'), + confirmCloseUnsaved: false, + hidePageHeader: true, + onClose: function(callbackData, customData) { + if (customData && reloadOnChange) { + location.reload(); + } else if (customData && trigger) { + trigger.trigger('attachments:updated'); + } + }, + }); + }; + + global.reloadManagementAttachmentInfoColumn = function reloadManagementAttachmentInfoColumn( + itemLocator, + column + ) { + $.ajax({ + url: build_url(Indico.Urls.ManagementAttachmentInfoColumn, itemLocator), + method: 'GET', + error: handleAjaxError, + success: function(data) { + column.replaceWith(data.html); + }, + }); + }; + + global.messageIfFolderProtected = function messageIfFolderProtected( + protectionField, + folderField, + protectionInfo, + selfProtection, + inheritedProtection, + folderProtection + ) { + folderField.on('change', function() { + var selectedFolder = $(this); + if (protectionInfo[selectedFolder.val()] && !protectionField.prop('checked')) { + selfProtection.hide(); + inheritedProtection.hide(); + folderProtection + .find('.folder-name') + .html(selectedFolder.children('option:selected').text()); + folderProtection.show(); + } else { + folderProtection.hide(); + selfProtection.toggle(protectionField.prop('checked')); + inheritedProtection.toggle(!protectionField.prop('checked')); + } + }); + _.defer(function() { + folderField.triggerHandler('change'); + }); + }; + + global.setupAttachmentTooltipButtons = function setupAttachmentTooltipButtons() { + $('.attachments-tooltip-button').each(function() { + var button = $(this); + button.qtip({ + content: { + text: button.next('.material_list'), + }, + show: { + event: 'click', + }, + hide: { + event: 'unfocus', + }, + position: { + my: 'top right', + at: 'bottom left', + }, + events: { + show: function() { + button.addClass('open'); + }, + hide: function() { + button.removeClass('open'); + }, + }, + style: { + classes: 'material_tip', + }, + }); + }); + }; +})(window); diff --git a/indico/modules/attachments/client/js/package.js b/indico/modules/attachments/client/js/package.js new file mode 100644 index 00000000000..40e9baf3b1e --- /dev/null +++ b/indico/modules/attachments/client/js/package.js @@ -0,0 +1,60 @@ +// This file is part of Indico. +// Copyright (C) 2002 - 2021 CERN +// +// Indico is free software; you can redistribute it and/or +// modify it under the terms of the MIT License; see the +// LICENSE file for more details. + +/* global handleFlashes:false, handleAjaxError:false */ + +import packageStatusURL from 'indico-url:attachments.package_status'; + +import {indicoAxios, handleAxiosError} from 'indico/utils/axios'; +import {$T} from 'indico/utils/i18n'; + +window.setupGeneratePackage = function setupGeneratePackage(eventId) { + $('#filter_type input:radio').on('change', function() { + $('#form-group-sessions').toggle(this.value === 'sessions'); + $('#form-group-contributions').toggle(this.value === 'contributions'); + $('#form-group-dates').toggle(this.value === 'dates'); + }); + $('#filter_type input:radio:checked').trigger('change'); + + const $form = $('#download-package-form'); + let closeLoader; + + async function poll(taskId) { + let res; + try { + res = await indicoAxios.get(packageStatusURL({event_id: eventId, task_id: taskId})); + } catch (error) { + handleAxiosError(error); + closeLoader(); + return; + } + if (res.data.download_url) { + window.location.href = res.data.download_url; + closeLoader(); + } else { + poll(taskId); + } + } + + $form.ajaxForm({ + error(...args) { + closeLoader(); + handleAjaxError(...args); + }, + beforeSubmit() { + closeLoader = IndicoUI.Dialogs.Util.progress($T.gettext('Building package')); + }, + success(data) { + if (data.success) { + poll(data.task_id); + } else { + handleFlashes(data, true, $form.find('.flashed-messages')); + closeLoader(); + } + }, + }); +}; diff --git a/indico/modules/attachments/clone.py b/indico/modules/attachments/clone.py index ef0901b521a..796b6f720c3 100644 --- a/indico/modules/attachments/clone.py +++ b/indico/modules/attachments/clone.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.orm import joinedload, subqueryload from indico.core.db import db diff --git a/indico/modules/attachments/controllers/compat.py b/indico/modules/attachments/controllers/compat.py index 0fc300e8487..2a3680c26c9 100644 --- a/indico/modules/attachments/controllers/compat.py +++ b/indico/modules/attachments/controllers/compat.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import current_app, redirect, request from werkzeug.exceptions import NotFound @@ -22,7 +20,7 @@ def _clean_args(kwargs): if 'event_id' not in kwargs: raise NotFound if is_legacy_id(kwargs['event_id']): - mapping = LegacyEventMapping.find(legacy_event_id=kwargs['event_id']).first_or_404() + mapping = LegacyEventMapping.query.filter_by(legacy_event_id=kwargs['event_id']).first_or_404() kwargs['event_id'] = mapping.event_id if 'contrib_id' in kwargs: kwargs['contribution_id'] = kwargs.pop('contrib_id') @@ -38,7 +36,7 @@ def _clean_args(kwargs): @RHSimple.wrap_function def compat_folder(**kwargs): _clean_args(kwargs) - folder = LegacyAttachmentFolderMapping.find(**kwargs).first_or_404().folder + folder = LegacyAttachmentFolderMapping.query.filter_by(**kwargs).first_or_404().folder if folder.is_deleted: raise NotFound return redirect(url_for('attachments.list_folder', folder), 302 if current_app.debug else 301) @@ -50,21 +48,20 @@ def compat_folder_old(): 'contribId': 'contrib_id', 'subContId': 'subcontrib_id', 'materialId': 'material_id'} - kwargs = {mapping[k]: v for k, v in request.args.iteritems() if k in mapping} + kwargs = {mapping[k]: v for k, v in request.args.items() if k in mapping} return compat_folder(**kwargs) def _redirect_to_note(**kwargs): del kwargs['material_id'] del kwargs['resource_id'] - kwargs['confId'] = kwargs.pop('event_id') return redirect(url_for('event_notes.view', **kwargs), 302 if current_app.debug else 301) @RHSimple.wrap_function def compat_attachment(**kwargs): _clean_args(kwargs) - mapping = LegacyAttachmentMapping.find_first(**kwargs) + mapping = LegacyAttachmentMapping.query.filter_by(**kwargs).first() if mapping is None: if kwargs['material_id'] == 'minutes' and kwargs['resource_id'] == 'minutes': return _redirect_to_note(**kwargs) diff --git a/indico/modules/attachments/controllers/display/base.py b/indico/modules/attachments/controllers/display/base.py index 7658fcf4d09..285e6a9883c 100644 --- a/indico/modules/attachments/controllers/display/base.py +++ b/indico/modules/attachments/controllers/display/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import redirect, request, session from werkzeug.exceptions import BadRequest, Forbidden @@ -20,7 +18,7 @@ class DownloadAttachmentMixin(SpecificAttachmentMixin): - """Download an attachment""" + """Download an attachment.""" def _check_access(self): if not self.attachment.can_access(session.user): diff --git a/indico/modules/attachments/controllers/display/category.py b/indico/modules/attachments/controllers/display/category.py index c532982b938..2742fa153cb 100644 --- a/indico/modules/attachments/controllers/display/category.py +++ b/indico/modules/attachments/controllers/display/category.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.attachments.controllers.display.base import DownloadAttachmentMixin from indico.modules.categories.controllers.base import RHDisplayCategoryBase diff --git a/indico/modules/attachments/controllers/display/event.py b/indico/modules/attachments/controllers/display/event.py index 820c278e93d..14f5c148ece 100644 --- a/indico/modules/attachments/controllers/display/event.py +++ b/indico/modules/attachments/controllers/display/event.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import redirect, request, session from werkzeug.exceptions import Forbidden @@ -25,8 +23,16 @@ def _process_args(self): DownloadAttachmentMixin._process_args(self) def _check_access(self): - RHDisplayEventBase._check_access(self) - DownloadAttachmentMixin._check_access(self) + try: + DownloadAttachmentMixin._check_access(self) + except Forbidden: + # if we get here the user has no access to the attachment itself so we + # trigger the event access check since it may show the access key form + # or registration required message + RHDisplayEventBase._check_access(self) + # the user may have access to the event but not the material so if we + # are here we need to re-raise the original exception + raise class RHListEventAttachmentFolder(SpecificFolderMixin, RHDisplayEventBase): @@ -35,8 +41,10 @@ def _process_args(self): SpecificFolderMixin._process_args(self) def _check_access(self): - RHDisplayEventBase._check_access(self) if not self.folder.can_access(session.user): + # basically the same logic as in RHDownloadEventAttachment. see the comments + # there for a more detailed explanation. + RHDisplayEventBase._check_access(self) raise Forbidden def _process(self): diff --git a/indico/modules/attachments/controllers/display/event_test.py b/indico/modules/attachments/controllers/display/event_test.py new file mode 100644 index 00000000000..47438252421 --- /dev/null +++ b/indico/modules/attachments/controllers/display/event_test.py @@ -0,0 +1,132 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from io import BytesIO + +import pytest +from flask import session +from werkzeug.exceptions import Forbidden + +from indico.core.db.sqlalchemy.protection import ProtectionMode +from indico.modules.attachments.controllers.display.event import RHDownloadEventAttachment +from indico.modules.attachments.models.attachments import Attachment, AttachmentFile, AttachmentType +from indico.modules.attachments.models.folders import AttachmentFolder +from indico.modules.events.controllers.base import AccessKeyRequired, RegistrationRequired +from indico.util.date_time import now_utc + + +pytest_plugins = 'indico.modules.events.registration.testing.fixtures' + + +def _make_attachment(user, obj): + folder = AttachmentFolder(title='dummy_folder', description='a dummy folder') + file = AttachmentFile(user=user, filename='dummy_file.txt', content_type='text/plain') + attachment = Attachment(folder=folder, user=user, title='dummy_attachment', type=AttachmentType.file, file=file) + attachment.folder.object = obj + attachment.file.save(BytesIO(b'hello world')) + return attachment + + +@pytest.fixture +def attachment_access_test_env(request_context, dummy_user, dummy_event, dummy_session, create_contribution): + session.set_session_user(dummy_user) + session_contrib = create_contribution(dummy_event, 'Session Contrib', session=dummy_session) + standalone_contrib = create_contribution(dummy_event, 'Standalone Contrib') + event_attachment = _make_attachment(dummy_user, dummy_event) + session_attachment = _make_attachment(dummy_user, dummy_session) + session_contrib_attachment = _make_attachment(dummy_user, session_contrib) + standalone_contrib_attachment = _make_attachment(dummy_user, standalone_contrib) + + rh = RHDownloadEventAttachment() + rh.event = dummy_event + + def assert_access_check(attachment, accessible=True, expected_exc=Forbidden): + __tracebackhide__ = True + rh.attachment = attachment + if accessible: + rh._check_access() + else: + with pytest.raises(expected_exc) as exc_info: + rh._check_access() + assert exc_info.type is expected_exc + + yield type('AttachmentAccessTestEnv', (object,), { + 'event': dummy_event, + 'session': dummy_session, + 'standalone_contrib': standalone_contrib, + 'session_contrib': session_contrib, + 'event_attachment': event_attachment, + 'session_attachment': session_attachment, + 'session_contrib_attachment': session_contrib_attachment, + 'standalone_contrib_attachment': standalone_contrib_attachment, + 'assert_access_check': staticmethod(assert_access_check), + }) + + +@pytest.mark.parametrize( + ('event_prot', 'session_prot', 'session_contrib_prot', 'standalone_contrib_prot', 'accessible'), ( + # Everything should be accessible in a public event + ('public', None, None, None, ('event', 'session', 'standalone_contrib', 'session_contrib')), + # Protecting the event should restrict everything + ('protected', None, None, None, ()), + # A public session in a protected event should allow access to its contents + ('protected', 'public', None, None, ('session', 'session_contrib')), + # Session access should not work if the contribution is protected + ('protected', 'public', 'protected', None, ('session',)), + # Public contribution access should be possible + ('protected', 'protected', 'public', 'public', ('session_contrib', 'standalone_contrib')), + ) +) +def test_access_checks(db, attachment_access_test_env, event_prot, session_prot, session_contrib_prot, + standalone_contrib_prot, accessible): + env = attachment_access_test_env + env.event.protection_mode = ProtectionMode[event_prot] + env.session.protection_mode = ProtectionMode[session_prot or 'inheriting'] + env.session_contrib.protection_mode = ProtectionMode[session_contrib_prot or 'inheriting'] + env.standalone_contrib.protection_mode = ProtectionMode[standalone_contrib_prot or 'inheriting'] + db.session.flush() + env.assert_access_check(env.event_attachment, 'event' in accessible) + env.assert_access_check(env.session_attachment, 'session' in accessible) + env.assert_access_check(env.standalone_contrib_attachment, 'standalone_contrib' in accessible) + env.assert_access_check(env.session_contrib_attachment, 'session_contrib' in accessible) + + +def test_self_protected(db, attachment_access_test_env): + env = attachment_access_test_env + env.event_attachment.protection_mode = ProtectionMode.protected + env.session_attachment.protection_mode = ProtectionMode.protected + env.standalone_contrib_attachment.protection_mode = ProtectionMode.protected + env.session_contrib_attachment.protection_mode = ProtectionMode.protected + db.session.flush() + env.assert_access_check(env.event_attachment, False) + env.assert_access_check(env.session_attachment, False) + env.assert_access_check(env.standalone_contrib_attachment, False) + env.assert_access_check(env.session_contrib_attachment, False) + + +def test_access_key(db, attachment_access_test_env): + env = attachment_access_test_env + env.event.protection_mode = ProtectionMode.protected + env.event.access_key = 'secret' + db.session.flush() + env.assert_access_check(env.event_attachment, False, AccessKeyRequired) + env.assert_access_check(env.session_attachment, False, AccessKeyRequired) + env.assert_access_check(env.standalone_contrib_attachment, False, AccessKeyRequired) + env.assert_access_check(env.session_contrib_attachment, False, AccessKeyRequired) + + +def test_only_registered(db, dummy_regform, attachment_access_test_env): + env = attachment_access_test_env + env.event.protection_mode = ProtectionMode.protected + env.event.public_regform_access = True + env.event.update_principal(dummy_regform, read_access=True) + dummy_regform.start_dt = now_utc() + db.session.flush() + env.assert_access_check(env.event_attachment, False, RegistrationRequired) + env.assert_access_check(env.session_attachment, False, RegistrationRequired) + env.assert_access_check(env.standalone_contrib_attachment, False, RegistrationRequired) + env.assert_access_check(env.session_contrib_attachment, False, RegistrationRequired) diff --git a/indico/modules/attachments/controllers/event_package.py b/indico/modules/attachments/controllers/event_package.py index 5e8c1a9591e..840c8947224 100644 --- a/indico/modules/attachments/controllers/event_package.py +++ b/indico/modules/attachments/controllers/event_package.py @@ -1,33 +1,36 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os -from collections import OrderedDict -from flask import flash, session +from celery.exceptions import TimeoutError +from flask import flash, jsonify, request, session from markupsafe import escape from sqlalchemy import Date, cast +from indico.core.celery import AsyncResult from indico.core.db import db from indico.core.db.sqlalchemy.links import LinkType +from indico.core.errors import IndicoError from indico.modules.attachments.forms import AttachmentPackageForm from indico.modules.attachments.models.attachments import Attachment, AttachmentFile, AttachmentType from indico.modules.attachments.models.folders import AttachmentFolder +from indico.modules.attachments.tasks import generate_materials_package from indico.modules.events.contributions.models.contributions import Contribution from indico.modules.events.contributions.models.subcontributions import SubContribution +from indico.modules.events.controllers.base import RHDisplayEventBase from indico.modules.events.sessions.models.sessions import Session from indico.modules.events.util import ZipGeneratorMixin from indico.util.date_time import format_date, format_time from indico.util.fs import secure_filename from indico.util.i18n import _ -from indico.util.string import natural_sort_key, to_unicode +from indico.util.string import natural_sort_key from indico.web.forms.base import FormDefaults +from indico.web.util import jsonify_data def _get_start_dt(obj): @@ -78,9 +81,10 @@ def _get_all_attachments(self, added_since): return [attachment for attachment in query if _get_start_dt(attachment.folder.object) is not None] def _build_base_query(self, added_since=None): - query = Attachment.find(Attachment.type == AttachmentType.file, ~AttachmentFolder.is_deleted, - ~Attachment.is_deleted, AttachmentFolder.event == self.event, - _join=AttachmentFolder) + query = (Attachment.query + .filter(Attachment.type == AttachmentType.file, ~AttachmentFolder.is_deleted, + ~Attachment.is_deleted, AttachmentFolder.event == self.event) + .join(AttachmentFolder)) if added_since is not None: query = query.join(Attachment.file).filter(cast(AttachmentFile.created_dt, Date) >= added_since) return query @@ -121,9 +125,9 @@ def _check_date(attachment): start_dt = _get_start_dt(attachment.folder.object) if start_dt is None: return None - return unicode(start_dt.date()) in dates + return str(start_dt.date()) in dates - return filter(_check_date, self._build_base_query()) + return list(filter(_check_date, self._build_base_query())) def _iter_items(self, attachments): for attachment in attachments: @@ -137,13 +141,13 @@ def _prepare_folder_structure(self, item): segments.append('Unscheduled') segments.extend(self._get_base_path(attachment)) if not attachment.folder.is_default: - segments.append(secure_filename(attachment.folder.title, unicode(attachment.folder.id))) + segments.append(secure_filename(attachment.folder.title, str(attachment.folder.id))) segments.append(attachment.file.filename) - path = os.path.join(*self._adjust_path_length(filter(None, segments))) + path = os.path.join(*self._adjust_path_length([_f for _f in segments if _f])) while path in self.used_filenames: # prepend the id if there's a path collision - segments[-1] = '{}-{}'.format(attachment.id, segments[-1]) - path = os.path.join(*self._adjust_path_length(filter(None, segments))) + segments[-1] = f'{attachment.id}-{segments[-1]}' + path = os.path.join(*self._adjust_path_length([_f for _f in segments if _f])) return path def _get_base_path(self, attachment): @@ -154,15 +158,15 @@ def _get_base_path(self, attachment): start_date = _get_start_dt(obj) if start_date is not None: if isinstance(obj, SubContribution): - paths.append(secure_filename('{}_{}'.format(obj.position, obj.title), '')) + paths.append(secure_filename(f'{obj.position}_{obj.title}', '')) else: time = format_time(start_date, format='HHmm', timezone=self.event.timezone) - paths.append(secure_filename('{}_{}'.format(to_unicode(time), obj.title), '')) + paths.append(secure_filename(f'{time}_{obj.title}', '')) else: if isinstance(obj, SubContribution): - paths.append(secure_filename('{}_{}'.format(obj.position, obj.title), unicode(obj.id))) + paths.append(secure_filename(f'{obj.position}_{obj.title}', str(obj.id))) else: - paths.append(secure_filename(obj.title, unicode(obj.id))) + paths.append(secure_filename(obj.title, str(obj.id))) obj = _get_obj_parent(obj) linked_obj_start_date = _get_start_dt(linked_object) @@ -179,22 +183,28 @@ class AttachmentPackageMixin(AttachmentPackageGeneratorMixin): def _process(self): form = self._prepare_form() if form.validate_on_submit(): - attachments = self._filter_attachments(form.data) + attachments = [attachment.id for attachment in self._filter_attachments(form.data)] if attachments: - return self._generate_zip_file(attachments) + task = generate_materials_package.delay(attachments, self.event) + return jsonify(task_id=task.id, success=True) else: flash(_('There are no materials matching your criteria.'), 'warning') + return jsonify_data(success=False) + elif form.is_submitted(): + flash('; '.join(form.error_list), 'warning') + return jsonify_data(success=False) return self.wp.render_template('generate_package.html', self.event, form=form, management=self.management) def _prepare_form(self): form = AttachmentPackageForm(obj=FormDefaults(filter_type='all')) form.dates.choices = list(self._iter_event_days()) - filter_types = OrderedDict() - filter_types['all'] = _('Everything') - filter_types['sessions'] = _('Specific sessions') - filter_types['contributions'] = _('Specific contributions') - filter_types['dates'] = _('Specific days') + filter_types = { + 'all': _('Everything'), + 'sessions': _('Specific sessions'), + 'contributions': _('Specific contributions'), + 'dates': _('Specific days'), + } form.sessions.choices = self._load_session_data() if not form.sessions.choices: @@ -206,7 +216,7 @@ def _prepare_form(self): del filter_types['contributions'] del form.contributions - form.filter_type.choices = filter_types.items() + form.filter_type.choices = list(filter_types.items()) return form def _load_session_data(self): @@ -228,4 +238,20 @@ def _format_contrib(contrib): def _iter_event_days(self): for day in self.event.iter_days(): - yield day.isoformat(), format_date(day, 'short').decode('utf-8') + yield day.isoformat(), format_date(day, 'short') + + +class RHPackageEventAttachmentsStatus(RHDisplayEventBase): + def _process(self): + res = AsyncResult(request.view_args['task_id']) + try: + download_url = res.get(5, propagate=False) + except TimeoutError: + return jsonify(download_url=None) + try: + if res.successful(): + return jsonify(download_url=download_url) + else: + raise IndicoError(_('Material package generation failed')) + finally: + res.forget() diff --git a/indico/modules/attachments/controllers/management/base.py b/indico/modules/attachments/controllers/management/base.py index 2ff708e184c..c987285103b 100644 --- a/indico/modules/attachments/controllers/management/base.py +++ b/indico/modules/attachments/controllers/management/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import mimetypes from flask import flash, render_template, request, session @@ -23,7 +21,6 @@ from indico.modules.attachments.util import get_attached_items from indico.util.fs import secure_client_filename from indico.util.i18n import _, ngettext -from indico.util.string import to_unicode from indico.web.flask.templating import get_template_module from indico.web.flask.util import url_for from indico.web.forms.base import FormDefaults @@ -56,12 +53,12 @@ def _get_parent_info(parent): def _get_folders_protection_info(linked_object): - folders = AttachmentFolder.find(object=linked_object, is_deleted=False) + folders = AttachmentFolder.query.filter_by(object=linked_object, is_deleted=False) return {folder.id: folder.is_self_protected for folder in folders} class ManageAttachmentsMixin: - """Shows the attachment management page""" + """Show the attachment management page.""" wp = None def _process(self): @@ -76,7 +73,7 @@ def _process(self): class AddAttachmentFilesMixin: - """Upload file attachments""" + """Upload file attachments.""" def _process(self): form = AddAttachmentFilesForm(linked_object=self.object) @@ -85,7 +82,7 @@ def _process(self): folder = form.folder.data or AttachmentFolder.get_or_create_default(linked_object=self.object) for f in files: filename = secure_client_filename(f.filename) - attachment = Attachment(folder=folder, user=session.user, title=to_unicode(f.filename), + attachment = Attachment(folder=folder, user=session.user, title=f.filename, type=AttachmentType.file, protection_mode=form.protection_mode.data) if attachment.is_self_protected: attachment.acl = form.acl.data @@ -101,11 +98,12 @@ def _process(self): return jsonify_data(attachment_list=_render_attachment_list(self.object)) return jsonify_template('attachments/upload.html', form=form, action=url_for('.upload', self.object), protection_message=_render_protection_message(self.object), - folders_protection_info=_get_folders_protection_info(self.object)) + folders_protection_info=_get_folders_protection_info(self.object), + existing_attachment=None) class AddAttachmentLinkMixin: - """Add link attachment""" + """Add link attachment.""" def _process(self): form = AddAttachmentLinkForm(linked_object=self.object) @@ -119,7 +117,7 @@ def _process(self): class EditAttachmentMixin(SpecificAttachmentMixin): - """Edit an attachment""" + """Edit an attachment.""" def _process(self): defaults = FormDefaults(self.attachment, protected=self.attachment.is_self_protected, skip_attrs={'file'}) @@ -158,7 +156,7 @@ def _process(self): class CreateFolderMixin: - """Create a new empty folder""" + """Create a new empty folder.""" def _process(self): form = AttachmentFolderForm(obj=FormDefaults(is_always_visible=True), linked_object=self.object) @@ -177,7 +175,7 @@ def _process(self): class EditFolderMixin(SpecificFolderMixin): - """Edit a folder""" + """Edit a folder.""" def _process(self): defaults = FormDefaults(self.folder, protected=self.folder.is_self_protected) @@ -197,7 +195,7 @@ def _process(self): class DeleteFolderMixin(SpecificFolderMixin): - """Delete a folder""" + """Delete a folder.""" def _process(self): self.folder.is_deleted = True @@ -208,7 +206,7 @@ def _process(self): class DeleteAttachmentMixin(SpecificAttachmentMixin): - """Delete an attachment""" + """Delete an attachment.""" def _process(self): self.attachment.is_deleted = True diff --git a/indico/modules/attachments/controllers/management/category.py b/indico/modules/attachments/controllers/management/category.py index c61e75c614e..7d38de952db 100644 --- a/indico/modules/attachments/controllers/management/category.py +++ b/indico/modules/attachments/controllers/management/category.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from werkzeug.exceptions import Forbidden diff --git a/indico/modules/attachments/controllers/management/event.py b/indico/modules/attachments/controllers/management/event.py index 0b26bb38333..5de416aeaa8 100644 --- a/indico/modules/attachments/controllers/management/event.py +++ b/indico/modules/attachments/controllers/management/event.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import jsonify, session from werkzeug.exceptions import Forbidden, NotFound diff --git a/indico/modules/attachments/controllers/util.py b/indico/modules/attachments/controllers/util.py index a9f9d767ba3..e0f87a32e0a 100644 --- a/indico/modules/attachments/controllers/util.py +++ b/indico/modules/attachments/controllers/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request from werkzeug.exceptions import NotFound @@ -15,7 +13,7 @@ class SpecificAttachmentMixin: - """Mixin for RHs that reference a specific attachment""" + """Mixin for RHs that reference a specific attachment.""" normalize_url_spec = { 'args': { @@ -30,13 +28,13 @@ class SpecificAttachmentMixin: } def _process_args(self): - self.attachment = Attachment.find_one(id=request.view_args['attachment_id'], is_deleted=False) + self.attachment = Attachment.query.filter_by(id=request.view_args['attachment_id'], is_deleted=False).one() if self.attachment.folder.is_deleted: raise NotFound class SpecificFolderMixin: - """Mixin for RHs that reference a specific folder""" + """Mixin for RHs that reference a specific folder.""" normalize_url_spec = { 'locators': { @@ -46,4 +44,4 @@ class SpecificFolderMixin: } def _process_args(self): - self.folder = AttachmentFolder.find_one(id=request.view_args['folder_id'], is_deleted=False) + self.folder = AttachmentFolder.query.filter_by(id=request.view_args['folder_id'], is_deleted=False).one() diff --git a/indico/modules/attachments/forms.py b/indico/modules/attachments/forms.py index 9fa4cebcc1b..29a5d5d0c1d 100644 --- a/indico/modules/attachments/forms.py +++ b/indico/modules/attachments/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms.ext.sqlalchemy.fields import QuerySelectField from wtforms.fields import BooleanField, TextAreaField from wtforms.fields.html5 import URLField @@ -35,15 +33,14 @@ class AttachmentFormBase(IndicoForm): allow_groups=True, allow_external_users=True, allow_event_roles=True, allow_category_roles=True, allow_registration_forms=True, event=lambda form: form.event, - default_text=_('Restrict access to this material'), description=_("The list of users and groups allowed to access the material")) def __init__(self, *args, **kwargs): linked_object = kwargs.pop('linked_object') self.event = getattr(linked_object, 'event', None) # not present in categories - super(AttachmentFormBase, self).__init__(*args, **kwargs) - self.folder.query = (AttachmentFolder - .find(object=linked_object, is_default=False, is_deleted=False) + super().__init__(*args, **kwargs) + self.folder.query = (AttachmentFolder.query + .filter_by(object=linked_object, is_default=False, is_deleted=False) .order_by(db.func.lower(AttachmentFolder.title))) @generated_data @@ -75,7 +72,7 @@ class EditAttachmentFileForm(EditAttachmentFormBase): description=_("Already uploaded file. Replace it by adding a new file.")) -class AttachmentLinkFormMixin(object): +class AttachmentLinkFormMixin: title = StringField(_("Title"), [DataRequired()]) link_url = URLField(_("URL"), [DataRequired()]) @@ -97,7 +94,6 @@ class AttachmentFolderForm(IndicoForm): allow_groups=True, allow_external_users=True, allow_event_roles=True, allow_category_roles=True, allow_registration_forms=True, event=lambda form: form.event, - default_text=_('Restrict access to this folder'), description=_("The list of users and groups allowed to access the folder")) is_always_visible = BooleanField(_("Always Visible"), [HiddenUnless('is_hidden', value=False)], @@ -115,13 +111,13 @@ class AttachmentFolderForm(IndicoForm): def __init__(self, *args, **kwargs): self.linked_object = kwargs.pop('linked_object') self.event = getattr(self.linked_object, 'event', None) # not present in categories - super(AttachmentFolderForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.title.choices = self._get_title_suggestions() def _get_title_suggestions(self): query = db.session.query(AttachmentFolder.title).filter_by(is_deleted=False, is_default=False, object=self.linked_object) - existing = set(x[0] for x in query) + existing = {x[0] for x in query} suggestions = set(get_default_folder_names()) - existing if self.title.data: suggestions.add(self.title.data) diff --git a/indico/modules/attachments/logging.py b/indico/modules/attachments/logging.py index 19bcdcfc06a..83c7d138b30 100644 --- a/indico/modules/attachments/logging.py +++ b/indico/modules/attachments/logging.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import wraps from jinja2.filters import do_filesizeformat @@ -28,7 +26,7 @@ def connect_log_signals(): def _ignore_non_loggable(f): """ - Only calls the decorated function the attachment/folder is not + Only call the decorated function if the attachment/folder is not linked to a category. """ @wraps(f) @@ -49,7 +47,7 @@ def _get_folder_data(folder, for_attachment=False): def _get_attachment_data(attachment): data = _get_folder_data(attachment.folder, True) - data['Type'] = unicode(attachment.type.title) + data['Type'] = str(attachment.type.title) data['Title'] = attachment.title if attachment.type == AttachmentType.link: data['URL'] = attachment.link_url @@ -67,37 +65,37 @@ def _log(event, kind, msg, user, data): @_ignore_non_loggable def _log_folder_created(folder, user, **kwargs): event = folder.object.event - _log(event, EventLogKind.positive, 'Created folder "{}"'.format(folder.title), user, _get_folder_data(folder)) + _log(event, EventLogKind.positive, f'Created folder "{folder.title}"', user, _get_folder_data(folder)) @_ignore_non_loggable def _log_folder_deleted(folder, user, **kwargs): event = folder.object.event - _log(event, EventLogKind.negative, 'Deleted folder "{}"'.format(folder.title), user, _get_folder_data(folder)) + _log(event, EventLogKind.negative, f'Deleted folder "{folder.title}"', user, _get_folder_data(folder)) @_ignore_non_loggable def _log_folder_updated(folder, user, **kwargs): event = folder.object.event - _log(event, EventLogKind.change, 'Updated folder "{}"'.format(folder.title), user, _get_folder_data(folder)) + _log(event, EventLogKind.change, f'Updated folder "{folder.title}"', user, _get_folder_data(folder)) @_ignore_non_loggable def _log_attachment_created(attachment, user, **kwargs): event = attachment.folder.object.event - _log(event, EventLogKind.positive, 'Added material "{}"'.format(attachment.title), user, + _log(event, EventLogKind.positive, f'Added material "{attachment.title}"', user, _get_attachment_data(attachment)) @_ignore_non_loggable def _log_attachment_deleted(attachment, user, **kwargs): event = attachment.folder.object.event - _log(event, EventLogKind.negative, 'Deleted material "{}"'.format(attachment.title), user, + _log(event, EventLogKind.negative, f'Deleted material "{attachment.title}"', user, _get_attachment_data(attachment)) @_ignore_non_loggable def _log_attachment_updated(attachment, user, **kwargs): event = attachment.folder.object.event - _log(event, EventLogKind.change, 'Updated material "{}"'.format(attachment.title), user, + _log(event, EventLogKind.change, f'Updated material "{attachment.title}"', user, _get_attachment_data(attachment)) diff --git a/indico/modules/attachments/models/attachments.py b/indico/modules/attachments/models/attachments.py index 88fd531e586..3329e213b2c 100644 --- a/indico/modules/attachments/models/attachments.py +++ b/indico/modules/attachments/models/attachments.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import posixpath from flask import g @@ -23,10 +21,10 @@ from indico.modules.attachments.preview import get_file_previewer from indico.modules.attachments.util import can_manage_attachments from indico.util.date_time import now_utc +from indico.util.enum import RichIntEnum from indico.util.fs import secure_filename from indico.util.i18n import _ -from indico.util.string import return_ascii, strict_unicode -from indico.util.struct.enum import RichIntEnum +from indico.util.string import strict_str from indico.web.flask.util import url_for @@ -85,26 +83,25 @@ def _build_storage_path(self): assert folder.object is not None if folder.link_type == LinkType.category: # category//... - path_segments = ['category', strict_unicode(folder.category.id)] + path_segments = ['category', strict_str(folder.category.id)] else: # event//event/... - path_segments = ['event', strict_unicode(folder.event.id), folder.link_type.name] + path_segments = ['event', strict_str(folder.event.id), folder.link_type.name] if folder.link_type == LinkType.session: # event//session//... - path_segments.append(strict_unicode(folder.session.id)) + path_segments.append(strict_str(folder.session.id)) elif folder.link_type == LinkType.contribution: # event//contribution//... - path_segments.append(strict_unicode(folder.contribution.id)) + path_segments.append(strict_str(folder.contribution.id)) elif folder.link_type == LinkType.subcontribution: # event//subcontribution//... - path_segments.append(strict_unicode(folder.subcontribution.id)) + path_segments.append(strict_str(folder.subcontribution.id)) self.attachment.assign_id() self.assign_id() filename = '{}-{}-{}'.format(self.attachment.id, self.id, secure_filename(self.filename, 'file')) path = posixpath.join(*(path_segments + [filename])) return config.ATTACHMENT_STORAGE, path - @return_ascii def __repr__(self): return ''.format( self.id, @@ -118,7 +115,7 @@ class Attachment(ProtectionMixin, VersionedResourceMixin, db.Model): __tablename__ = 'attachments' __table_args__ = ( # links: url but no file - db.CheckConstraint('type != {} OR (link_url IS NOT NULL AND file_id IS NULL)'.format(AttachmentType.link.value), + db.CheckConstraint(f'type != {AttachmentType.link.value} OR (link_url IS NOT NULL AND file_id IS NULL)', 'valid_link'), # we can't require the file_id to be NOT NULL for files because of the circular relationship... # but we can ensure that we never have both a file_id AND a link_url...for @@ -224,7 +221,7 @@ def locator(self): return dict(self.folder.locator, attachment_id=self.id) def get_download_url(self, absolute=False): - """Returns the download url for the attachment. + """Return the download url for the attachment. During static site generation this returns a local URL for the file or the target URL for the link. @@ -239,24 +236,23 @@ def get_download_url(self, absolute=False): @property def download_url(self): - """The download url for the attachment""" + """The download url for the attachment.""" return self.get_download_url() @property def absolute_download_url(self): - """The absolute download url for the attachment""" + """The absolute download url for the attachment.""" return self.get_download_url(absolute=True) def can_access(self, user, *args, **kwargs): - """Checks if the user is allowed to access the attachment. + """Check if the user is allowed to access the attachment. This is the case if the user has access to see the attachment or if the user can manage attachments for the linked object. """ - return (super(Attachment, self).can_access(user, *args, **kwargs) or + return (super().can_access(user, *args, **kwargs) or can_manage_attachments(self.folder.object, user)) - @return_ascii def __repr__(self): return ''.format( self.id, @@ -279,11 +275,11 @@ def _offline_download_url(attachment): if isinstance(attachment.folder.object, db.m.Event): path = "" elif isinstance(attachment.folder.object, db.m.Session): - path = "{}-session".format(attachment.folder.session.friendly_id) + path = f"{attachment.folder.session.friendly_id}-session" elif isinstance(attachment.folder.object, db.m.Contribution): - path = "{}-contribution".format(attachment.folder.contribution.friendly_id) + path = f"{attachment.folder.contribution.friendly_id}-contribution" elif isinstance(attachment.folder.object, db.m.SubContribution): - path = "{}-subcontribution".format(attachment.folder.subcontribution.friendly_id) + path = f"{attachment.folder.subcontribution.friendly_id}-subcontribution" else: return '' return posixpath.join("material", path, str(attachment.id) + "-" + attachment.file.filename) diff --git a/indico/modules/attachments/models/folders.py b/indico/modules/attachments/models/folders.py index af24a4d4081..bc21edf3469 100644 --- a/indico/modules/attachments/models/folders.py +++ b/indico/modules/attachments/models/folders.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import defaultdict from flask import g @@ -24,7 +22,6 @@ from indico.modules.attachments.util import can_manage_attachments from indico.util.decorators import strict_classproperty from indico.util.locators import locator_property -from indico.util.string import return_ascii class AttachmentFolder(LinkMixin, ProtectionMixin, db.Model): @@ -38,7 +35,7 @@ class AttachmentFolder(LinkMixin, ProtectionMixin, db.Model): @strict_classproperty @staticmethod def __auto_table_args(): - default_inheriting = 'not (is_default and protection_mode != {})'.format(ProtectionMode.inheriting.value) + default_inheriting = f'not (is_default and protection_mode != {ProtectionMode.inheriting.value})' return (db.CheckConstraint(default_inheriting, 'default_inheriting'), db.CheckConstraint('is_default = (title IS NULL)', 'default_or_title'), db.CheckConstraint('not (is_default and is_deleted)', 'default_not_deleted'), @@ -118,15 +115,15 @@ def protection_parent(self): @classmethod def get_or_create_default(cls, linked_object): - """Gets the default folder for the given object or creates it.""" - folder = cls.find_first(is_default=True, object=linked_object) + """Get the default folder for the given object or creates it.""" + folder = cls.query.filter_by(is_default=True, object=linked_object).first() if folder is None: folder = cls(is_default=True, object=linked_object) return folder @classmethod def get_or_create(cls, linked_object, title=None): - """Gets a folder for the given object or creates it. + """Get a folder for the given object or create it. If no folder title is specified, the default folder will be used. It is the caller's responsibility to add the folder @@ -136,7 +133,8 @@ def get_or_create(cls, linked_object, title=None): if title is None: return AttachmentFolder.get_or_create_default(linked_object) else: - folder = AttachmentFolder.find_first(object=linked_object, is_default=False, is_deleted=False, title=title) + folder = AttachmentFolder.query.filter_by(object=linked_object, is_default=False, is_deleted=False, + title=title).first() return folder or AttachmentFolder(object=linked_object, title=title) @locator_property @@ -144,16 +142,16 @@ def locator(self): return dict(self.object.locator, folder_id=self.id) def can_access(self, user, *args, **kwargs): - """Checks if the user is allowed to access the folder. + """Check if the user is allowed to access the folder. This is the case if the user has access the folder or if the user can manage attachments for the linked object. """ - return (super(AttachmentFolder, self).can_access(user, *args, **kwargs) or + return (super().can_access(user, *args, **kwargs) or can_manage_attachments(self.object, user)) def can_view(self, user): - """Checks if the user can see the folder. + """Check if the user can see the folder. This does not mean the user can actually access its contents. It just determines if it is visible to him or not. @@ -162,11 +160,11 @@ def can_view(self, user): return False if not self.object.can_access(user): return False - return self.is_always_visible or super(AttachmentFolder, self).can_access(user) + return self.is_always_visible or super().can_access(user) @classmethod def get_for_linked_object(cls, linked_object, preload_event=False): - """Gets the attachments for the given object. + """Get the attachments for the given object. This only returns attachments that haven't been deleted. @@ -207,7 +205,6 @@ def get_for_linked_object(cls, linked_object, preload_event=False): return g.event_attachments[event].get(linked_object, []) - @return_ascii def __repr__(self): return ''.format( self.id, diff --git a/indico/modules/attachments/models/folders_test.py b/indico/modules/attachments/models/folders_test.py index 71309414a40..b998c1d58cb 100644 --- a/indico/modules/attachments/models/folders_test.py +++ b/indico/modules/attachments/models/folders_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.attachments import AttachmentFolder diff --git a/indico/modules/attachments/models/legacy_mapping.py b/indico/modules/attachments/models/legacy_mapping.py index 7c94222c428..0a061ba593c 100644 --- a/indico/modules/attachments/models/legacy_mapping.py +++ b/indico/modules/attachments/models/legacy_mapping.py @@ -1,20 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db from indico.core.db.sqlalchemy.util.models import auto_table_args -from indico.util.string import return_ascii -class _LegacyLinkMixin(object): +class _LegacyLinkMixin: events_backref_name = None @declared_attr @@ -60,14 +57,14 @@ def event(cls): @property def link_repr(self): - """A kwargs-style string suitable for the object's repr""" + """A kwargs-style string suitable for the object's repr.""" _all_columns = {'event_id', 'contribution_id', 'subcontribution_id', 'session_id'} info = [(key, getattr(self, key)) for key in _all_columns if getattr(self, key) is not None] - return ', '.join('{}={}'.format(key, value) for key, value in info) + return ', '.join(f'{key}={value}' for key, value in info) class LegacyAttachmentFolderMapping(_LegacyLinkMixin, db.Model): - """Legacy attachmentfolder id mapping + """Legacy attachmentfolder id mapping. Legacy folders ("materials") had ids unique only within their linked object. This table maps those ids for a specific object @@ -97,7 +94,6 @@ def __table_args__(cls): backref=db.backref('legacy_mapping', uselist=False, lazy=True) ) - @return_ascii def __repr__(self): return ''.format( self.folder, self.material_id, self.link_repr @@ -105,7 +101,7 @@ def __repr__(self): class LegacyAttachmentMapping(_LegacyLinkMixin, db.Model): - """Legacy attachment id mapping + """Legacy attachment id mapping. Legacy attachments ("resources") had ids unique only within their folder and its linked object. This table maps those ids for a @@ -139,7 +135,6 @@ def __table_args__(cls): backref=db.backref('legacy_mapping', uselist=False, lazy=True) ) - @return_ascii def __repr__(self): return ''.format( self.attachment, self.material_id, self.resource_id, self.link_repr diff --git a/indico/modules/attachments/models/principals.py b/indico/modules/attachments/models/principals.py index bb7985b28db..872b7b72ccd 100644 --- a/indico/modules/attachments/models/principals.py +++ b/indico/modules/attachments/models/principals.py @@ -1,18 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db from indico.core.db.sqlalchemy.principals import PrincipalMixin from indico.core.db.sqlalchemy.util.models import auto_table_args -from indico.util.string import return_ascii class AttachmentFolderPrincipal(PrincipalMixin, db.Model): @@ -42,9 +39,8 @@ def __table_args__(cls): # relationship backrefs: # - folder (AttachmentFolder.acl_entries) - @return_ascii def __repr__(self): - return ''.format(self.id, self.folder_id, self.principal) + return f'' class AttachmentPrincipal(PrincipalMixin, db.Model): @@ -74,6 +70,5 @@ def __table_args__(cls): # relationship backrefs: # - attachment (Attachment.acl_entries) - @return_ascii def __repr__(self): - return ''.format(self.id, self.attachment_id, self.principal) + return f'' diff --git a/indico/modules/attachments/operations.py b/indico/modules/attachments/operations.py index 28784b0beb5..8615870fdfa 100644 --- a/indico/modules/attachments/operations.py +++ b/indico/modules/attachments/operations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals @@ -17,7 +15,7 @@ def add_attachment_link(data, linked_object): - """Add a link attachment to linked_object""" + """Add a link attachment to linked_object.""" folder = data.pop('folder', None) if not folder: folder = AttachmentFolder.get_or_create_default(linked_object=linked_object) diff --git a/indico/modules/attachments/preview.py b/indico/modules/attachments/preview.py index d69a3381c75..84384f53a09 100644 --- a/indico/modules/attachments/preview.py +++ b/indico/modules/attachments/preview.py @@ -1,41 +1,40 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import re from flask import render_template, session from indico.core import signals from indico.util.signals import values_from_signal -from indico.util.string import fix_broken_string -class Previewer(object): - """Base class for file previewers +class Previewer: + """Base class for file previewers. To create a new file prewiewer, subclass this class and register it using the `get_file_previewers` signal. """ + ALLOWED_CONTENT_TYPE = None TEMPLATES_DIR = 'attachments/previewers/' TEMPATE = None @classmethod def can_preview(cls, attachment_file): - """Checks if the content type of the file matches the allowed content + """ + Check if the content type of the file matches the allowed content type of files that the previewer can be used for. """ return cls.ALLOWED_CONTENT_TYPE.search(attachment_file.content_type) is not None @classmethod def generate_content(cls, attachment): - """Generates the HTML output of the file preview""" + """Generate the HTML output of the file preview.""" return render_template(cls.TEMPLATES_DIR + cls.TEMPLATE, attachment=attachment) @@ -50,7 +49,7 @@ class PDFPreviewer(Previewer): @classmethod def can_preview(cls, attachment_file): - if not super(PDFPreviewer, cls).can_preview(attachment_file) or not session.user: + if not super().can_preview(attachment_file) or not session.user: return False return session.user.settings.get('use_previewer_pdf', False) @@ -71,12 +70,22 @@ class TextPreviewer(Previewer): @classmethod def generate_content(cls, attachment): with attachment.file.open() as f: - return render_template(cls.TEMPLATES_DIR + 'text_preview.html', attachment=attachment, - text=fix_broken_string(f.read(), as_unicode=True)) + content = f.read() + try: + text = content.decode() # utf-8 + except UnicodeDecodeError: + try: + text = content.decode('latin1') + except UnicodeDecodeError: + # not sure if there's anything where latin1 decoding fails, but just in + # case we decode such a file as utf-8 and replace anything that's invalid + text = content.decode(errors='replace') + return render_template(cls.TEMPLATES_DIR + 'text_preview.html', attachment=attachment, text=text) def get_file_previewer(attachment_file): - """Returns a file previewer for the given attachment file based on the + """ + Return a file previewer for the given attachment file based on the file's content type. """ for previewer in get_file_previewers(): diff --git a/indico/modules/attachments/tasks.py b/indico/modules/attachments/tasks.py new file mode 100644 index 00000000000..234760a5633 --- /dev/null +++ b/indico/modules/attachments/tasks.py @@ -0,0 +1,26 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from indico.core.celery import celery +from indico.core.db import db +from indico.modules.attachments.models.attachments import Attachment +from indico.modules.files.models.files import File + + +@celery.task(ignore_result=False) +def generate_materials_package(attachment_ids, event): + from indico.modules.attachments.controllers.event_package import AttachmentPackageGeneratorMixin + attachments = Attachment.query.filter(Attachment.id.in_(attachment_ids)).all() + attachment_package_mixin = AttachmentPackageGeneratorMixin() + attachment_package_mixin.event = event + generated_zip = attachment_package_mixin._generate_zip_file(attachments, return_file=True) + f = File(filename='material-package.zip', content_type='application/zip', meta={'event_id': event.id}) + context = ('event', event.id, 'attachment-package') + f.save(context, generated_zip) + db.session.add(f) + db.session.commit() + return f.signed_download_url diff --git a/indico/modules/attachments/templates/_attachments.html b/indico/modules/attachments/templates/_attachments.html index f7af822c54c..9a8e054a1dd 100644 --- a/indico/modules/attachments/templates/_attachments.html +++ b/indico/modules/attachments/templates/_attachments.html @@ -8,10 +8,12 @@ {% macro render_attachment(attachment, has_label=false, classes='') %} {% set previewable = (attachment.file.is_previewable and not g.static_site) or false %} + {% set visible_link = attachment.type.name == 'link' and (not attachment.is_protected or attachment.can_access(session.user)) %} {%- if not has_label -%} diff --git a/indico/modules/attachments/templates/_display.html b/indico/modules/attachments/templates/_display.html index e129989bd28..357c580aabe 100644 --- a/indico/modules/attachments/templates/_display.html +++ b/indico/modules/attachments/templates/_display.html @@ -14,10 +14,12 @@

{% endmacro %} - - -{% macro render_attachments_button(files, folders) %} - - -{% endmacro %} diff --git a/indico/modules/attachments/templates/generate_package.html b/indico/modules/attachments/templates/generate_package.html index 9c4628c399c..34764afe1ab 100644 --- a/indico/modules/attachments/templates/generate_package.html +++ b/indico/modules/attachments/templates/generate_package.html @@ -45,7 +45,8 @@ {% block content %} {% call wrap_in_ibox(management) %} - {{ form_header(form, disable_if_locked=false) }} + {{ form_header(form, disable_if_locked=false, id='download-package-form') }} +
{{ form_rows(form) }} {% call form_footer(form) %} @@ -53,15 +54,6 @@ {% endcall %} {% endblock %} diff --git a/indico/modules/attachments/util.py b/indico/modules/attachments/util.py index b4d832c5f96..02eaca192ae 100644 --- a/indico/modules/attachments/util.py +++ b/indico/modules/attachments/util.py @@ -1,20 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core.db import db def get_attached_folders(linked_object, include_empty=True, include_hidden=True, preload_event=False): - """ - Return a list of all the folders linked to an object. + """Return a list of all the folders linked to an object. :param linked_object: The object whose attachments are to be returned :param include_empty: Whether to return empty folders as well. @@ -62,33 +59,8 @@ def get_attached_items(linked_object, include_empty=True, include_hidden=True, p } -def get_nested_attached_items(obj): - """ - Returns a structured representation of all attachments linked to an object - and all its nested objects. - - :param obj: A :class:`Event`, :class:`Session`, :class:`Contribution` - or :class:`SubContribution` object. - """ - attachments = get_attached_items(obj, include_empty=False, include_hidden=False) - nested_objects = [] - if isinstance(obj, db.m.Event): - nested_objects = obj.sessions + obj.contributions - elif isinstance(obj, db.m.Session): - nested_objects = obj.contributions - elif isinstance(obj, db.m.Contribution): - nested_objects = obj.subcontributions - if nested_objects: - children = filter(None, map(get_nested_attached_items, nested_objects)) - if children: - attachments['children'] = children - if attachments: - attachments['object'] = obj - return attachments - - def can_manage_attachments(obj, user): - """Checks if a user can manage attachments for the object""" + """Check if a user can manage attachments for the object.""" if not user: return False if obj.can_manage(user): diff --git a/indico/modules/attachments/views.py b/indico/modules/attachments/views.py index 0b02c1de31d..e4f758fa179 100644 --- a/indico/modules/attachments/views.py +++ b/indico/modules/attachments/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement from indico.modules.events.views import WPConferenceDisplayBase, WPSimpleEventDisplayBase from indico.web.views import WPJinjaMixin diff --git a/indico/modules/auth/__init__.py b/indico/modules/auth/__init__.py index 4b0abc032dd..8db34b5d26d 100644 --- a/indico/modules/auth/__init__.py +++ b/indico/modules/auth/__init__.py @@ -1,19 +1,19 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import redirect, request, session from flask_multipass import MultipassException +from werkzeug.exceptions import Forbidden from indico.core import signals from indico.core.auth import multipass from indico.core.config import config from indico.core.db import db +from indico.core.errors import NoReportError from indico.core.logger import Logger from indico.modules.auth.models.identities import Identity from indico.modules.auth.models.registration_requests import RegistrationRequest @@ -75,7 +75,7 @@ def process_identity(identity_info): def login_user(user, identity=None, admin_impersonation=False): - """Set the session user and performs on-login logic + """Set the session user and performs on-login logic. When specifying `identity`, the provider/identitifer information is saved in the session so the identity management page can prevent @@ -91,7 +91,7 @@ def login_user(user, identity=None, admin_impersonation=False): session.timezone = user.settings.get('timezone', config.DEFAULT_TIMEZONE) else: session.timezone = 'LOCAL' - session.user = user + session.set_session_user(user) session.lang = user.settings.get('lang') if not admin_impersonation: if identity: @@ -100,6 +100,7 @@ def login_user(user, identity=None, admin_impersonation=False): else: session.pop('login_identity', None) user.synchronize_data() + signals.users.logged_in.send(user, identity=identity, admin_impersonation=admin_impersonation) @signals.menu.items.connect_via('user-profile-sidemenu') @@ -109,7 +110,33 @@ def _extend_profile_sidemenu(sender, user, **kwargs): @signals.users.registered.connect def _delete_requests(user, **kwargs): - for req in RegistrationRequest.find(RegistrationRequest.email.in_(user.all_emails)): + for req in RegistrationRequest.query.filter(RegistrationRequest.email.in_(user.all_emails)): logger.info('Deleting registration request %r due to registration of %r', req, user) db.session.delete(req) db.session.flush() + + +@signals.app_created.connect +def _handle_insecure_password_logins(app, **kwargs): + @app.before_request + def _redirect_if_insecure(): + if not request.endpoint: + return + + if ( + request.blueprint == 'assets' or + request.endpoint.endswith('.static') or + request.endpoint in ('auth.logout', 'auth.accounts', 'core.contact', 'core.change_lang') + ): + return + + if 'insecure_password_error' not in session: + return + + if request.method != 'GET': + raise NoReportError.wrap_exc(Forbidden(_('You need to change your password'))) + + if request.is_xhr or request.is_json: + return + + return redirect(url_for('auth.accounts')) diff --git a/indico/modules/auth/blueprint.py b/indico/modules/auth/blueprint.py index 53b835c11fb..3f7a54daf79 100644 --- a/indico/modules/auth/blueprint.py +++ b/indico/modules/auth/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request from indico.modules.auth.controllers import (RHAccounts, RHAdminImpersonate, RHLinkAccount, RHLogin, RHLoginForm, diff --git a/indico/modules/auth/controllers.py b/indico/modules/auth/controllers.py index 6c88e57d0ac..69e6d628d12 100644 --- a/indico/modules/auth/controllers.py +++ b/indico/modules/auth/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, jsonify, redirect, render_template, request, session from itsdangerous import BadData, BadSignature from markupsafe import Markup @@ -14,7 +12,7 @@ from werkzeug.exceptions import BadRequest, Forbidden, NotFound from indico.core import signals -from indico.core.auth import multipass +from indico.core.auth import login_rate_limiter, multipass from indico.core.config import config from indico.core.db import db from indico.core.notifications import make_email, send_email @@ -49,7 +47,7 @@ def _get_provider(name, external): class RHLogin(RH): - """The login page""" + """The login page.""" # Disable global CSRF check. The form might not be an IndicoForm # but a normal WTForm from Flask-WTF which does not use the same @@ -65,9 +63,14 @@ def _process(self): multipass.set_next_url() return multipass.redirect_success() + # Some clients attempt to incorrectly resolve redirections internally. + # See https://github.com/indico/indico/issues/4720 for details + user_agent = request.headers.get('User-Agent', '') + sso_redirect = not any(s in user_agent for s in ('ms-office', 'Microsoft Office')) + # If we have only one provider, and this provider is external, we go there immediately # However, after a failed login we need to show the page to avoid a redirect loop - if not session.pop('_multipass_auth_failed', False) and 'provider' not in request.view_args: + if not session.pop('_multipass_auth_failed', False) and 'provider' not in request.view_args and sso_redirect: single_auth_provider = multipass.single_auth_provider if single_auth_provider and single_auth_provider.is_external: multipass.set_next_url() @@ -82,25 +85,30 @@ def _process(self): return provider.initiate_external_login() # If we have a POST request we submitted a login form for a local provider + rate_limit_exceeded = False if request.method == 'POST': active_provider = provider = _get_provider(request.form['_provider'], False) form = provider.login_form() - if form.validate_on_submit(): + rate_limit_exceeded = not login_rate_limiter.test() + if not rate_limit_exceeded and form.validate_on_submit(): response = multipass.handle_login_form(provider, form.data) if response: return response + # re-check since a failed login may have triggered the rate limit + rate_limit_exceeded = not login_rate_limiter.test() # Otherwise we show the form for the default provider else: active_provider = multipass.default_local_auth_provider form = active_provider.login_form() if active_provider else None - providers = multipass.auth_providers.values() + providers = list(multipass.auth_providers.values()) + retry_in = login_rate_limiter.get_reset_delay() if rate_limit_exceeded else None return render_template('auth/login_page.html', form=form, providers=providers, active_provider=active_provider, - login_reason=login_reason) + login_reason=login_reason, retry_in=retry_in) class RHLoginForm(RH): - """Retrieves a login form (json)""" + """Retrieve a login form (json).""" def _process(self): provider = _get_provider(request.view_args['provider'], False) @@ -110,10 +118,13 @@ def _process(self): class RHLogout(RH): - """Logs the user out""" + """Log the user out.""" def _process(self): - return multipass.logout(request.args.get('next') or url_for_index(), clear_session=True) + next_url = request.args.get('next') + if not next_url or not multipass.validate_next_url(next_url): + next_url = url_for_index() + return multipass.logout(next_url, clear_session=True) def _send_confirmation(email, salt, endpoint, template, template_args=None, url_args=None, data=None): @@ -129,7 +140,7 @@ def _send_confirmation(email, salt, endpoint, template, template_args=None, url_ class RHLinkAccount(RH): - """Links a new identity with an existing user. + """Link a new identity with an existing user. This RH is only used if the identity information contains an email address and an existing user was found. @@ -161,7 +172,7 @@ def _process(self): if self.must_choose_email: form = SelectEmailForm() - form.email.choices = zip(self.emails, self.emails) + form.email.choices = list(zip(self.emails, self.emails)) else: form = IndicoForm() @@ -196,7 +207,7 @@ def _send_confirmation(self, email): class RHRegister(RH): - """Creates a new indico user. + """Create a new indico user. This handles two cases: - creation of a new user with a locally stored username and password @@ -220,7 +231,7 @@ def _process_args(self): raise Forbidden('Local registration is disabled') def _get_verified_email(self): - """Checks if there is an email verification token.""" + """Check if there is an email verification token.""" try: token = request.args['token'] except KeyError: @@ -282,8 +293,8 @@ def _send_confirmation(self, email): def _prepare_registration_data(self, form, handler): email = form.email.data extra_emails = handler.get_all_emails(form) - {email} - user_data = {k: v for k, v in form.data.viewitems() if k in {'first_name', 'last_name', 'affiliation', - 'address', 'phone'}} + user_data = {k: v for k, v in form.data.items() + if k in {'first_name', 'last_name', 'affiliation', 'address', 'phone'}} user_data.update(handler.get_extra_user_data(form)) identity_data = handler.get_identity_data(form) settings = { @@ -317,7 +328,7 @@ def _create_user(self, form, handler): class RHAccounts(RHUserBase): - """Displays user accounts""" + """Display user accounts.""" def _create_form(self): if self.user.local_identity: @@ -336,9 +347,12 @@ def _handle_edit_local_account(self, form): self.user.local_identity.identifier = form.data['username'] if form.data['new_password']: self.user.local_identity.password = form.data['new_password'] + session.pop('insecure_password_error', None) flash(_("Your local account credentials have been updated successfully"), 'success') def _process(self): + insecure_login_password_error = session.get('insecure_password_error') + form = self._create_form() if form.validate_on_submit(): if isinstance(form, AddLocalIdentityForm): @@ -346,13 +360,14 @@ def _process(self): elif isinstance(form, EditLocalIdentityForm): self._handle_edit_local_account(form) return redirect(url_for('auth.accounts')) - provider_titles = {name: provider.title for name, provider in multipass.identity_providers.iteritems()} + provider_titles = {name: provider.title for name, provider in multipass.identity_providers.items()} return WPAuthUser.render_template('accounts.html', 'accounts', - form=form, user=self.user, provider_titles=provider_titles) + form=form, user=self.user, provider_titles=provider_titles, + insecure_login_password_error=insecure_login_password_error) class RHRemoveAccount(RHUserBase): - """Removes an identity linked to a user""" + """Remove an identity linked to a user.""" def _process_args(self): RHUserBase._process_args(self) @@ -375,7 +390,7 @@ def _process(self): return redirect(url_for('.accounts')) -class RegistrationHandler(object): +class RegistrationHandler: form = None def __init__(self, rh): @@ -445,18 +460,18 @@ def get_form_defaults(self): return FormDefaults(self.identity_info['data']) def create_form(self): - form = super(MultipassRegistrationHandler, self).create_form() + form = super().create_form() # We only want the phone/address fields if the provider gave us data for it for field in {'address', 'phone'}: if field in form and not self.identity_info['data'].get(field): delattr(form, field) emails = self.identity_info['data'].getlist('email') - form.email.choices = zip(emails, emails) + form.email.choices = list(zip(emails, emails)) return form def form(self, **kwargs): if self.from_sync_provider: - synced_values = {k: v or '' for k, v in self.identity_info['data'].iteritems()} + synced_values = {k: v or '' for k, v in self.identity_info['data'].items()} return MultipassRegistrationForm(synced_fields=multipass.synced_fields, synced_values=synced_values, **kwargs) else: @@ -471,7 +486,7 @@ def moderate_registrations(self): return self.identity_info['moderated'] def get_all_emails(self, form): - emails = super(MultipassRegistrationHandler, self).get_all_emails(form) + emails = super().get_all_emails(form) return emails | set(self.identity_info['data'].getlist('email')) def get_identity_data(self, form): @@ -480,7 +495,7 @@ def get_identity_data(self, form): 'data': self.identity_info['data'], 'multipass_data': self.identity_info['multipass_data']} def get_extra_user_data(self, form): - data = super(MultipassRegistrationHandler, self).get_extra_user_data(form) + data = super().get_extra_user_data(form) if self.from_sync_provider: data['synced_fields'] = form.synced_fields | {field for field in multipass.synced_fields if field not in form} @@ -494,8 +509,9 @@ class LocalRegistrationHandler(RegistrationHandler): form = LocalRegistrationForm def __init__(self, rh): - if 'next' in request.args: - session['register_next_url'] = request.args['next'] + next_url = request.args.get('next') + if next_url and multipass.validate_next_url(next_url): + session['register_next_url'] = next_url @property def widget_attrs(self): @@ -510,7 +526,7 @@ def moderate_registrations(self): return config.LOCAL_MODERATION def get_all_emails(self, form): - emails = super(LocalRegistrationHandler, self).get_all_emails(form) + emails = super().get_all_emails(form) if not self.must_verify_email: emails.add(session['register_verified_email']) return emails @@ -532,7 +548,7 @@ def get_form_defaults(self): return FormDefaults(**data) def create_form(self): - form = super(LocalRegistrationHandler, self).create_form() + form = super().create_form() if not self.must_verify_email: form.email.data = session['register_verified_email'] return form @@ -547,7 +563,7 @@ def redirect_success(self): class RHResetPassword(RH): - """Resets the password for a local identity.""" + """Reset the password for a local identity.""" def _process_args(self): if not config.LOCAL_IDENTITIES: diff --git a/indico/modules/auth/forms.py b/indico/modules/auth/forms.py index cf938404d1d..770b2df8249 100644 --- a/indico/modules/auth/forms.py +++ b/indico/modules/auth/forms.py @@ -1,21 +1,19 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms.fields import PasswordField, SelectField, StringField, TextAreaField from wtforms.fields.html5 import EmailField -from wtforms.validators import DataRequired, Email, Length, Optional, ValidationError +from wtforms.validators import DataRequired, Email, Optional, ValidationError from indico.modules.auth import Identity from indico.modules.users import User from indico.util.i18n import _ from indico.web.forms.base import IndicoForm, SyncedInputsMixin -from indico.web.forms.validators import ConfirmPassword, used_if_not_synced +from indico.web.forms.validators import ConfirmPassword, SecurePassword, used_if_not_synced from indico.web.forms.widgets import SyncedInputWidget @@ -40,19 +38,26 @@ class LocalLoginForm(IndicoForm): class AddLocalIdentityForm(IndicoForm): username = StringField(_('Username'), [DataRequired(), _check_existing_username], filters=[_tolower]) - password = PasswordField(_('Password'), [DataRequired(), Length(min=5)]) - confirm_password = PasswordField(_('Confirm password'), [DataRequired(), ConfirmPassword('password')]) + password = PasswordField(_('Password'), [DataRequired(), SecurePassword('set-user-password', + username_field='username')], + render_kw={'autocomplete': 'new-password'}) + confirm_password = PasswordField(_('Confirm password'), [DataRequired(), ConfirmPassword('password')], + render_kw={'autocomplete': 'new-password'}) class EditLocalIdentityForm(IndicoForm): username = StringField(_('Username'), [DataRequired()], filters=[_tolower]) - password = PasswordField(_('Current password'), [DataRequired()]) - new_password = PasswordField(_('New password'), [Optional(), Length(min=5)]) - confirm_new_password = PasswordField(_('Confirm password'), [ConfirmPassword('new_password')]) + password = PasswordField(_('Current password'), [DataRequired()], + render_kw={'autocomplete': 'current-password'}) + new_password = PasswordField(_('New password'), [Optional(), SecurePassword('set-user-password', + username_field='username')], + render_kw={'autocomplete': 'new-password'}) + confirm_new_password = PasswordField(_('Confirm password'), [ConfirmPassword('new_password')], + render_kw={'autocomplete': 'new-password'}) def __init__(self, *args, **kwargs): self.identity = kwargs.pop('identity', None) - super(EditLocalIdentityForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def validate_password(self, field): if field.data != self.identity.password: @@ -95,14 +100,17 @@ class MultipassRegistrationForm(SyncedInputsMixin, IndicoForm): class LocalRegistrationForm(RegistrationForm): email = EmailField(_('Email address'), [Email(), _check_existing_email]) username = StringField(_('Username'), [DataRequired(), _check_existing_username], filters=[_tolower]) - password = PasswordField(_('Password'), [DataRequired(), Length(min=5)]) - confirm_password = PasswordField(_('Confirm password'), [DataRequired(), ConfirmPassword('password')]) + password = PasswordField(_('Password'), [DataRequired(), SecurePassword('set-user-password', + username_field='username')], + render_kw={'autocomplete': 'new-password'}) + confirm_password = PasswordField(_('Confirm password'), [DataRequired(), ConfirmPassword('password')], + render_kw={'autocomplete': 'new-password'}) comment = TextAreaField(_('Comment'), description=_("You can provide additional information or a comment for the " "administrators who will review your registration.")) @property def data(self): - data = super(LocalRegistrationForm, self).data + data = super().data data.pop('confirm_password', None) return data @@ -129,5 +137,6 @@ def user(self): class ResetPasswordForm(IndicoForm): username = StringField(_('Username')) - password = PasswordField(_('New password'), [DataRequired(), Length(min=5)]) + password = PasswordField(_('New password'), [DataRequired(), SecurePassword('set-user-password', + username_field='username')]) confirm_password = PasswordField(_('Confirm password'), [DataRequired(), ConfirmPassword('password')]) diff --git a/indico/modules/auth/models/identities.py b/indico/modules/auth/models/identities.py index e37e557f8b3..35c8fb24c0c 100644 --- a/indico/modules/auth/models/identities.py +++ b/indico/modules/auth/models/identities.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import datetime from sqlalchemy.dialects.postgresql import INET, JSONB @@ -16,11 +14,10 @@ from indico.core.db.sqlalchemy import UTCDateTime from indico.util.date_time import as_utc, now_utc from indico.util.passwords import PasswordProperty -from indico.util.string import return_ascii class Identity(db.Model): - """Identities of Indico users""" + """Identities of Indico users.""" __tablename__ = 'identities' __table_args__ = (db.UniqueConstraint('provider', 'identifier'), {'schema': 'users'}) @@ -93,14 +90,13 @@ def locator(self): @property def safe_last_login_dt(self): - """last_login_dt that is safe for sorting (no None values)""" + """last_login_dt that is safe for sorting (no None values).""" return self.last_login_dt or as_utc(datetime(1970, 1, 1)) def register_login(self, ip): - """Updates the last login information""" + """Update the last login information.""" self.last_login_dt = now_utc() self.last_login_ip = ip - @return_ascii def __repr__(self): - return ''.format(self.id, self.user_id, self.provider, self.identifier) + return f'' diff --git a/indico/modules/auth/models/registration_requests.py b/indico/modules/auth/models/registration_requests.py index 8121ecbebae..d1a1214bd16 100644 --- a/indico/modules/auth/models/registration_requests.py +++ b/indico/modules/auth/models/registration_requests.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import ARRAY, JSONB from werkzeug.datastructures import MultiDict from indico.core.db import db from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class RegistrationRequest(db.Model): @@ -81,6 +79,5 @@ def identity_data(self, identity_data): identity_data['data'] = dict(identity_data['data'].lists()) self._identity_data = identity_data - @return_ascii def __repr__(self): return format_repr(self, 'id', 'email') diff --git a/indico/modules/auth/providers.py b/indico/modules/auth/providers.py index c77885a56b9..0413c086cf9 100644 --- a/indico/modules/auth/providers.py +++ b/indico/modules/auth/providers.py @@ -1,17 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - +from flask import session from flask_multipass.providers.sqlalchemy import SQLAlchemyAuthProviderBase, SQLAlchemyIdentityProviderBase from indico.modules.auth import Identity from indico.modules.auth.forms import LocalLoginForm from indico.modules.users import User +from indico.util.passwords import validate_secure_password class IndicoAuthProvider(SQLAlchemyAuthProviderBase): @@ -23,7 +23,11 @@ class IndicoAuthProvider(SQLAlchemyAuthProviderBase): def check_password(self, identity, password): # No, the passwords are not stored in plaintext. Magic is happening here! - return identity.password == password + if identity.password != password: + return False + if error := validate_secure_password('login', password, username=identity.identifier, fast=True): + session['insecure_password_error'] = error + return True class IndicoIdentityProvider(SQLAlchemyIdentityProviderBase): diff --git a/indico/modules/auth/templates/accounts.html b/indico/modules/auth/templates/accounts.html index 9e06cc6247c..6ecd7815d35 100644 --- a/indico/modules/auth/templates/accounts.html +++ b/indico/modules/auth/templates/accounts.html @@ -3,6 +3,19 @@ {% block user_content %} {% if indico_config.LOCAL_IDENTITIES %} + {% if insecure_login_password_error %} + {% call message_box('error') %} + {% trans %} + The password you used to login is no longer secure: + {% endtrans %} + {{ insecure_login_password_error }} +
+ {% trans %} + You need to change your password in order to keep using Indico. + {% endtrans %} + {% endcall %} + {% endif %} +
@@ -126,6 +139,11 @@ title="{% trans %}The account used to log in cannot be unlinked{% endtrans %}"> {% trans %}Unlink{% endtrans %} + {% elif insecure_login_password_error %} + {% else %}
{% endif %} {% include 'flashed_messages.html' %} + {% if retry_in %} + {% call message_box('error') %} + {% trans delay=retry_in|format_human_timedelta(granularity='minutes') %} + Too many failed login attempts. Please wait {{ delay }}. + {% endtrans %} + {% endcall %} + {% endif %}
{{ login_form(active_provider, form) }}
diff --git a/indico/modules/auth/util.py b/indico/modules/auth/util.py index 37b6dcc17a0..5e9ee69b69f 100644 --- a/indico/modules/auth/util.py +++ b/indico/modules/auth/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import redirect, request, session from werkzeug.datastructures import MultiDict @@ -14,12 +12,11 @@ from indico.modules.auth import Identity from indico.modules.users.operations import create_user from indico.util.signing import secure_serializer -from indico.util.string import to_unicode from indico.web.flask.util import url_for def save_identity_info(identity_info, user): - """Saves information from IdentityInfo in the session""" + """Save information from IdentityInfo in the session.""" trusted_email = identity_info.provider.settings.get('trusted_email', False) session['login_identity_info'] = { 'provider': identity_info.provider.name, @@ -34,7 +31,7 @@ def save_identity_info(identity_info, user): def load_identity_info(): - """Retrieves identity information from the session""" + """Retrieve identity information from the session.""" try: info = session['login_identity_info'].copy() except KeyError: @@ -48,7 +45,7 @@ def load_identity_info(): def register_user(email, extra_emails, user_data, identity_data, settings, from_moderation=False): """ - Create a user based on the registration data provided during te + Create a user based on the registration data provided during the user registration process (via `RHRegister` and `RegistrationHandler`). This method is not meant to be used for generic user creation, the @@ -62,15 +59,15 @@ def register_user(email, extra_emails, user_data, identity_data, settings, from_ def impersonate_user(user): - """Impersonate another user as an admin""" - from indico.modules.auth import login_user, logger + """Impersonate another user as an admin.""" + from indico.modules.auth import logger, login_user current_user = session.user # We don't overwrite a previous entry - the original (admin) user should be kept there # XXX: Don't change this to setdefault - building `session_data` pops stuff from the session if 'login_as_orig_user' not in session: session['login_as_orig_user'] = { - 'session_data': {k: session.pop(k) for k in session.keys() if k[0] != '_' or k in ('_timezone', '_lang')}, + 'session_data': {k: session.pop(k) for k in list(session) if k[0] != '_' or k in ('_timezone', '_lang')}, 'user_id': session.user.id, 'user_name': session.user.get_full_name(last_name_first=False, last_name_upper=False) } @@ -79,7 +76,7 @@ def impersonate_user(user): def undo_impersonate_user(): - """Undo an admin impersonation login and revert to the old user""" + """Undo an admin impersonation login and revert to the old user.""" from indico.modules.auth import logger from indico.modules.users import User @@ -90,12 +87,12 @@ def undo_impersonate_user(): return user = User.get_or_404(entry['user_id']) logger.info('Admin %r stopped impersonating user %r', user, session.user) - session.user = user + session.set_session_user(user) session.update(entry['session_data']) def redirect_to_login(next_url=None, reason=None): - """Redirects to the login page. + """Redirect to the login page. :param next_url: URL to be redirected upon successful login. If not specified, it will be set to ``request.relative_url``. @@ -104,7 +101,7 @@ def redirect_to_login(next_url=None, reason=None): if not next_url: next_url = request.relative_url if reason: - session['login_reason'] = to_unicode(reason) + session['login_reason'] = reason return redirect(url_for_login(next_url)) diff --git a/indico/modules/auth/views.py b/indico/modules/auth/views.py index 18afedde0fe..88ff5abd9ca 100644 --- a/indico/modules/auth/views.py +++ b/indico/modules/auth/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.users.views import WPUser from indico.web.views import WPDecorated, WPJinjaMixin diff --git a/indico/modules/bootstrap/blueprint.py b/indico/modules/bootstrap/blueprint.py index ec8d77ea5bd..635a112dde9 100644 --- a/indico/modules/bootstrap/blueprint.py +++ b/indico/modules/bootstrap/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.bootstrap.controllers import RHBootstrap from indico.web.flask.wrappers import IndicoBlueprint diff --git a/indico/modules/bootstrap/client/js/index.js b/indico/modules/bootstrap/client/js/index.js index 8d531a0e4e1..2f67a91ccf5 100644 --- a/indico/modules/bootstrap/client/js/index.js +++ b/indico/modules/bootstrap/client/js/index.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/bootstrap/controllers.py b/indico/modules/bootstrap/controllers.py index bd4070c2b2f..16fe91db1d9 100644 --- a/indico/modules/bootstrap/controllers.py +++ b/indico/modules/bootstrap/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from platform import python_version from flask import flash, redirect, render_template, request, session @@ -24,7 +22,6 @@ from indico.modules.users import User from indico.util.i18n import _, get_all_locales from indico.util.network import is_private_url -from indico.util.string import to_unicode from indico.util.system import get_os from indico.web.flask.templating import get_template_module from indico.web.flask.util import url_for @@ -56,10 +53,10 @@ def _process_POST(self): # Creating new user user = User() - user.first_name = to_unicode(setup_form.first_name.data) - user.last_name = to_unicode(setup_form.last_name.data) - user.affiliation = to_unicode(setup_form.affiliation.data) - user.email = to_unicode(setup_form.email.data) + user.first_name = setup_form.first_name.data + user.last_name = setup_form.last_name.data + user.affiliation = setup_form.affiliation.data + user.email = setup_form.email.data user.is_admin = True identity = Identity(provider='indico', identifier=setup_form.username.data, password=setup_form.password.data) @@ -90,13 +87,13 @@ def _process_POST(self): try: register_instance(contact_name, contact_email) except (HTTPError, ValueError) as err: - message = get_template_module('bootstrap/flash_messages.html').community_error(err=err) + message = get_template_module('bootstrap/flash_messages.html').community_error(err=str(err)) category = 'error' except Timeout: message = get_template_module('bootstrap/flash_messages.html').community_timeout() category = 'error' except RequestException as exc: - message = get_template_module('bootstrap/flash_messages.html').community_exception(exc=exc) + message = get_template_module('bootstrap/flash_messages.html').community_exception(err=str(exc)) category = 'error' else: message = get_template_module('bootstrap/flash_messages.html').community_success() diff --git a/indico/modules/bootstrap/forms.py b/indico/modules/bootstrap/forms.py index b842c61a75b..46cd535810e 100644 --- a/indico/modules/bootstrap/forms.py +++ b/indico/modules/bootstrap/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms import BooleanField, StringField from wtforms.fields.html5 import EmailField from wtforms.validators import DataRequired, Email diff --git a/indico/modules/bootstrap/templates/flash_messages.html b/indico/modules/bootstrap/templates/flash_messages.html index 7a123c8a3d9..9f047bc1cd4 100644 --- a/indico/modules/bootstrap/templates/flash_messages.html +++ b/indico/modules/bootstrap/templates/flash_messages.html @@ -20,15 +20,15 @@ {% endmacro %} {% macro community_error(err) %} - {%- trans error=err.message %}Failed to register to the Community Hub: {{ error }}.{% endtrans %}
+ {%- trans %}Failed to register to the Community Hub: {{ err }}.{% endtrans %}
{%- trans link=url_for('cephalopod.index') -%} See the logs for details and try again here. {%- endtrans -%} {% endmacro %} -{% macro community_exception(exc) %} - {%- trans error=exc.message -%} - An unexpected exception happened while registering the server to the Community Hub: {{ error }}. +{% macro community_exception(err) %} + {%- trans -%} + An unexpected exception happened while registering the server to the Community Hub: {{ err }}. {%- endtrans %}
{%- trans link=url_for('cephalopod.index') -%} See the logs for details and try again here. diff --git a/indico/modules/categories/__init__.py b/indico/modules/categories/__init__.py index 1eaccdfa059..454dabf73a6 100644 --- a/indico/modules/categories/__init__.py +++ b/indico/modules/categories/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals diff --git a/indico/modules/categories/blueprint.py b/indico/modules/categories/blueprint.py index c7ba8d357f2..f6a6c6a6cda 100644 --- a/indico/modules/categories/blueprint.py +++ b/indico/modules/categories/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import redirect, request from indico.modules.categories.compat import compat_category @@ -17,8 +15,7 @@ RHCategoryUpcomingEvent, RHDisplayCategory, RHEventList, RHExportCategoryAtom, RHExportCategoryICAL, RHReachableCategoriesInfo, RHShowFutureEventsInCategory, - RHShowPastEventsInCategory, RHSubcatInfo, - RHXMLExportCategoryInfo) + RHShowPastEventsInCategory, RHSubcatInfo) from indico.modules.categories.controllers.management import (RHAddCategoryRole, RHAddCategoryRoleMembers, RHCategoryRoleMembersImportCSV, RHCategoryRoles, RHCreateCategory, RHDeleteCategory, RHDeleteCategoryRole, @@ -34,7 +31,7 @@ def _redirect_event_creation(category_id, event_type): - anchor = 'create-event:{}:{}'.format(event_type, category_id) + anchor = f'create-event:{event_type}:{category_id}' return redirect(url_for('.display', category_id=category_id, _anchor=anchor)) @@ -95,9 +92,6 @@ def _redirect_event_creation(category_id, event_type): # Event creation - redirect to anchor page opening the dialog _bp.add_url_rule('/create/event/', view_func=_redirect_event_creation) -# TODO: remember to refactor it at some point -_bp.add_url_rule('!/xmlGateway.py/getCategoryInfo', 'category_xml_info', RHXMLExportCategoryInfo) - # Short URLs _bp.add_url_rule('!/categ/', view_func=redirect_view('.display'), strict_slashes=False) _bp.add_url_rule('!/c/', view_func=redirect_view('.display'), strict_slashes=False) diff --git a/indico/modules/categories/client/js/base.jsx b/indico/modules/categories/client/js/base.jsx new file mode 100644 index 00000000000..4f696273d16 --- /dev/null +++ b/indico/modules/categories/client/js/base.jsx @@ -0,0 +1,34 @@ +// This file is part of Indico. +// Copyright (C) 2002 - 2021 CERN +// +// Indico is free software; you can redistribute it and/or +// modify it under the terms of the MIT License; see the +// LICENSE file for more details. + +import React from 'react'; +import ReactDOM from 'react-dom'; + +import {IButton, ICSCalendarLink} from 'indico/react/components'; +import {Translate} from 'indico/react/i18n'; + +document.addEventListener('DOMContentLoaded', () => { + const calendarContainer = document.querySelector('#category-calendar-link'); + + if (!calendarContainer) { + return; + } + + const categoryId = calendarContainer.dataset.categoryId; + + ReactDOM.render( + ( + + )} + options={[{key: 'category', text: Translate.string('Category'), extraParams: {}}]} + />, + calendarContainer + ); +}); diff --git a/indico/modules/categories/client/js/calendar.js b/indico/modules/categories/client/js/calendar.js index e2443b020c7..0bebc31950f 100644 --- a/indico/modules/categories/client/js/calendar.js +++ b/indico/modules/categories/client/js/calendar.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/categories/client/js/components/CategoryStatistics.jsx b/indico/modules/categories/client/js/components/CategoryStatistics.jsx index 2a391ccc4cd..0239a64cf99 100644 --- a/indico/modules/categories/client/js/components/CategoryStatistics.jsx +++ b/indico/modules/categories/client/js/components/CategoryStatistics.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/categories/client/js/components/CategoryStatistics.module.scss b/indico/modules/categories/client/js/components/CategoryStatistics.module.scss index ef1e010a20a..931480dbd2c 100644 --- a/indico/modules/categories/client/js/components/CategoryStatistics.module.scss +++ b/indico/modules/categories/client/js/components/CategoryStatistics.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/categories/client/js/context.js b/indico/modules/categories/client/js/context.js index 00f4784e866..2949fd462e1 100644 --- a/indico/modules/categories/client/js/context.js +++ b/indico/modules/categories/client/js/context.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/categories/client/js/display.js b/indico/modules/categories/client/js/display.js index 3844b8919f9..e16ae31ac93 100644 --- a/indico/modules/categories/client/js/display.js +++ b/indico/modules/categories/client/js/display.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/categories/client/js/index.js b/indico/modules/categories/client/js/index.js index 76e111318b3..db434dd1027 100644 --- a/indico/modules/categories/client/js/index.js +++ b/indico/modules/categories/client/js/index.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -9,7 +9,10 @@ import React from 'react'; import ReactDOM from 'react-dom'; import './display'; +import './base'; + import {setMomentLocale} from 'indico/utils/date'; + import CategoryStatistics from './components/CategoryStatistics'; import {LocaleContext} from './context.js'; diff --git a/indico/modules/categories/client/js/management.js b/indico/modules/categories/client/js/management.js index 4af9ecfbd8e..ea08e0bf218 100644 --- a/indico/modules/categories/client/js/management.js +++ b/indico/modules/categories/client/js/management.js @@ -1,24 +1,28 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -/* global strnatcmp:false, paginatedSelectAll:false */ +/* global strnatcmp:false, paginatedSelectAll:false, handleAjaxError:false, cornerMessage:false, + enableIfChecked:false, build_url:false, ajaxDialog:false, updateHtml:false */ -(function(global) { - 'use strict'; +import _ from 'lodash'; + +import {showUserSearch} from 'indico/react/components/principals/imperative'; +import {$T} from 'indico/utils/i18n'; +(function(global) { // Category cache - var _categories = {}; + const _categories = {}; global.setupCategoryMoveButton = function setupCategoryMoveButton(parentCategoryId) { if (parentCategoryId) { _fetchSourceCategory(parentCategoryId); } $('.js-move-category').on('click', function() { - var $this = $(this); + const $this = $(this); _moveCategories( [$this.data('categoryId')], _categories[parentCategoryId], @@ -29,16 +33,16 @@ global.setupCategoryTable = function setupCategoryTable(categoryId) { _fetchSourceCategory(categoryId); - var $table = $('table.category-management'); - var $tbody = $table.find('tbody'); - var $bulkDeleteButton = $('.js-bulk-delete-category'); - var $bulkMoveButton = $('.js-bulk-move-category'); - var categoryRowSelector = 'tr[data-category-id]'; - var checkboxSelector = 'input[name=category_id]'; + const $table = $('table.category-management'); + const $tbody = $table.find('tbody'); + const $bulkDeleteButton = $('.js-bulk-delete-category'); + const $bulkMoveButton = $('.js-bulk-move-category'); + const categoryRowSelector = 'tr[data-category-id]'; + const checkboxSelector = 'input[name=category_id]'; $('.js-sort-categories').on('click', function() { - var sortOrder = $(this).data('sort-order'); - var currentOrder = getSortedCategories(); + const sortOrder = $(this).data('sort-order'); + const currentOrder = getSortedCategories(); function undo() { restoreCategoryOrder(currentOrder); return setOrderAjax(currentOrder); @@ -55,7 +59,7 @@ }); $bulkMoveButton.on('click', function(evt) { - var $this = $(this); + const $this = $(this); evt.preventDefault(); if ($this.hasClass('disabled')) { return; @@ -64,18 +68,18 @@ }); $table.find('.js-move-category').on('click', function() { - var $this = $(this); + const $this = $(this); _moveCategories([$this.data('categoryId')], _categories[categoryId], $this.data('href')); }); $table.find('.js-delete-category').on('indico:confirmed', function(evt) { evt.preventDefault(); - var $this = $(this); + const $this = $(this); $.ajax({ url: $this.data('href'), method: 'POST', error: handleAjaxError, - success: function(data) { + success(data) { $this.closest(categoryRowSelector).remove(); updateCategoryDeleteButton(data.is_parent_empty); }, @@ -107,7 +111,7 @@ handle: '.js-handle', items: '> tr', tolerance: 'pointer', - update: function() { + update() { setOrderAjax(getSortedCategories()); }, }); @@ -124,7 +128,7 @@ function restoreCategoryOrder(order) { $.each(order, function(index, id) { $tbody - .find('[data-category-id=' + id + ']') + .find(`[data-category-id=${id}]`) .not('.js-move-category') .detach() .appendTo($tbody); @@ -177,7 +181,7 @@ } function getBulkDeleteButtonTooltipContent() { - var $checked = getSelectedRows(); + const $checked = getSelectedRows(); if ($checked.length) { if ($bulkDeleteButton.hasClass('disabled')) { return $T.gettext( @@ -198,7 +202,7 @@ } function bulkDeleteCategories() { - var $selectedRows = getSelectedRows(); + const $selectedRows = getSelectedRows(); ajaxDialog({ url: $table.data('bulk-delete-url'), method: 'POST', @@ -206,7 +210,7 @@ data: { category_id: getSelectedCategories(), }, - onClose: function(data) { + onClose(data) { if (data && data.success) { // Prevent other categories from being selected when someone reloads // the page after deleting a selected category. @@ -219,7 +223,7 @@ } function getBulkMoveButtonTooltipContent() { - var $checked = getSelectedRows(); + const $checked = getSelectedRows(); if ($checked.length) { return $T .ngettext( @@ -236,7 +240,7 @@ function getSelectedCategories() { return $table .find(categoryRowSelector) - .find(checkboxSelector + ':checked') + .find(`${checkboxSelector}:checked`) .map(function() { return this.value; }) @@ -244,11 +248,13 @@ } function getSelectedRows() { - return $table.find(categoryRowSelector).has(checkboxSelector + ':checked'); + return $table.find(categoryRowSelector).has(`${checkboxSelector}:checked`); } }; global.setupCategoryEventList = function setupCategoryEventList(categoryId) { + let isEverythingSelected = false; + _fetchSourceCategory(categoryId); enableIfChecked('#event-management', 'input[name=event_id]', '.js-enabled-if-checked'); @@ -264,7 +270,7 @@ return; } - var data = {}; + const data = {}; if (isEverythingSelected()) { data.all_selected = 1; // do NOT change this to true - the code on the server expects '1' } else { @@ -279,19 +285,19 @@ _moveEvents(_categories[categoryId], $(this).data('href'), data); }); - var isEverythingSelected = paginatedSelectAll({ + isEverythingSelected = paginatedSelectAll({ containerSelector: '#event-management', checkboxSelector: 'input:checkbox[name=event_id]', allSelectedSelector: 'input:checkbox[name=all_selected]', selectionMessageSelector: '#selection-message', totalRows: $('#event-management').data('total'), messages: { - allSelected: function(total) { + allSelected(total) { return $T .ngettext('*', 'All {0} events in this category are currently selected.') .format(total); }, - pageSelected: function(selected, total) { + pageSelected(selected, total) { return $T .ngettext( 'Only {0} out of {1} events is currently selected.', @@ -310,13 +316,13 @@ $.ajax({ url: build_url(Indico.Urls.Categories.info, {category_id: categoryId}), dataType: 'json', - error: function(xhr) { + error(xhr) { // XXX: Re-enable error handling once we skip retrieving protected parents if (xhr.status && xhr.status !== 403) { handleAjaxError(xhr); } }, - success: function(data) { + success(data) { _categories[categoryId] = data; }, }); @@ -324,8 +330,8 @@ } function _moveCategories(ids, source, endpoint) { - var sourceId = _.isObject(source) ? source.category.id : source; - var data = {category_id: ids}; + const sourceId = _.isObject(source) ? source.category.id : source; + const data = {category_id: ids}; $('
').categorynavigator({ category: source, @@ -341,10 +347,9 @@ ) .format(ids.length), actionOn: { - categoriesWithEvents: {disabled: true}, categoriesDescendingFrom: { disabled: true, - ids: ids, + ids, }, categories: { disabled: true, @@ -358,7 +363,7 @@ ), }, { - ids: ids, + ids, message: $T.ngettext( 'This is the category you are trying to move', 'This is one of the categories you are trying to move', @@ -368,13 +373,13 @@ ], }, }, - onAction: function(category) { + onAction(category) { $.ajax({ url: endpoint, type: 'POST', data: $.extend({target_category_id: category.id}, data), error: handleAjaxError, - success: function(data) { + success(data) { if (data.success) { location.reload(); } @@ -385,8 +390,8 @@ } function _moveEvents(source, endpoint, data) { - var sourceId = _.isObject(source) ? source.category.id : source; - var eventCount = data + const sourceId = _.isObject(source) ? source.category.id : source; + const eventCount = data ? data.all_selected ? $('#event-management').data('total') : data.event_id.length @@ -404,9 +409,6 @@ eventCount ), actionOn: { - categoriesWithSubcategories: { - disabled: true, - }, categoriesWithoutEventCreationRights: { disabled: true, }, @@ -420,13 +422,13 @@ ), }, }, - onAction: function(category) { + onAction(category) { $.ajax({ url: endpoint, type: 'POST', data: $.extend({target_category_id: category.id}, data || {}), error: handleAjaxError, - success: function(data) { + success(data) { if (data.success) { location.reload(); } @@ -437,13 +439,13 @@ } function setupRolesToggle() { - var $roles = $('#event-roles'); + const $roles = $('#event-roles'); $roles.on('click', '.toggle-members', function() { - var $row = $(this) + const $row = $(this) .closest('tr') .next('tr') .find('.slide'); - $row.css('max-height', $row[0].scrollHeight + 'px'); + $row.css('max-height', `${$row[0].scrollHeight}px`); $row.toggleClass('open close'); }); @@ -451,34 +453,30 @@ $(this) .find('.slide') .each(function() { - $(this).css('max-height', this.scrollHeight + 'px'); + $(this).css('max-height', `${this.scrollHeight}px`); }); }); } function setupRolesButtons() { - $('#event-roles').on('click', '.js-add-members', function(evt) { + $('#event-roles').on('click', '.js-add-members', async evt => { evt.stopPropagation(); - var $this = $(this); - $('
') - .principalfield({ - multiChoice: true, - onAdd: function(users) { - $.ajax({ - url: $this.data('href'), - method: $this.data('method'), - data: JSON.stringify({users: users}), - dataType: 'json', - contentType: 'application/json', - error: handleAjaxError, - complete: IndicoUI.Dialogs.Util.progress(), - success: function(data) { - updateHtml($this.data('update'), data); - }, - }); + const $this = $(evt.target); + const users = await showUserSearch({withExternalUsers: true}); + if (users.length) { + $.ajax({ + url: $this.data('href'), + method: $this.data('method'), + data: JSON.stringify({users}), + dataType: 'json', + contentType: 'application/json', + error: handleAjaxError, + complete: IndicoUI.Dialogs.Util.progress(), + success(data) { + updateHtml($this.data('update'), data); }, - }) - .principalfield('choose'); + }); + } }); } diff --git a/indico/modules/categories/compat.py b/indico/modules/categories/compat.py index f998c26c7e5..203e4a77775 100644 --- a/indico/modules/categories/compat.py +++ b/indico/modules/categories/compat.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import re from flask import abort, redirect, request @@ -21,9 +19,9 @@ def compat_category(legacy_category_id, path=None): if not re.match(r'^\d+l\d+$', legacy_category_id): abort(404) - mapping = LegacyCategoryMapping.find_first(legacy_category_id=legacy_category_id) + mapping = LegacyCategoryMapping.query.filter_by(legacy_category_id=legacy_category_id).first() if mapping is None: - raise NotFound('Legacy category {} does not exist'.format(legacy_category_id)) + raise NotFound(f'Legacy category {legacy_category_id} does not exist') view_args = request.view_args.copy() view_args['legacy_category_id'] = mapping.category_id # To create the same URL with the proper ID we take advantage of the diff --git a/indico/modules/categories/controllers/admin.py b/indico/modules/categories/controllers/admin.py index 68eae3bacf6..eb2e6e71793 100644 --- a/indico/modules/categories/controllers/admin.py +++ b/indico/modules/categories/controllers/admin.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, redirect from indico.modules.admin import RHAdminBase diff --git a/indico/modules/categories/controllers/base.py b/indico/modules/categories/controllers/base.py index b96098f3b4f..666c1d68c94 100644 --- a/indico/modules/categories/controllers/base.py +++ b/indico/modules/categories/controllers/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request, session from werkzeug.exceptions import Forbidden, NotFound @@ -39,7 +37,7 @@ def _process_args(self): class RHDisplayCategoryBase(RHCategoryBase): - """Base class for category display pages""" + """Base class for category display pages.""" def _check_access(self): if not self.category.can_access(session.user): diff --git a/indico/modules/categories/controllers/display.py b/indico/modules/categories/controllers/display.py index a3537439fb6..33f40127544 100644 --- a/indico/modules/categories/controllers/display.py +++ b/indico/modules/categories/controllers/display.py @@ -1,22 +1,20 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import date, datetime, time, timedelta from functools import partial from io import BytesIO -from itertools import chain, groupby, imap +from itertools import chain, groupby from operator import attrgetter, itemgetter from time import mktime import dateutil from dateutil.relativedelta import relativedelta -from flask import Response, flash, jsonify, redirect, request, session +from flask import flash, jsonify, redirect, request, session from pytz import utc from sqlalchemy.orm import joinedload, load_only, subqueryload, undefer, undefer_group from werkzeug.exceptions import BadRequest, NotFound @@ -25,7 +23,9 @@ from indico.core.db.sqlalchemy.colors import ColorTuple from indico.core.db.sqlalchemy.util.queries import get_n_matching from indico.modules.categories.controllers.base import RHDisplayCategoryBase -from indico.modules.categories.legacy import XMLCategorySerializer +from indico.modules.categories.controllers.util import (get_category_view_params, group_by_month, + make_format_event_date_func, make_happening_now_func, + make_is_recent_func) from indico.modules.categories.models.categories import Category from indico.modules.categories.serialize import (serialize_categories_ical, serialize_category, serialize_category_atom, serialize_category_chain) @@ -33,7 +33,6 @@ from indico.modules.categories.views import WPCategory, WPCategoryCalendar, WPCategoryStatistics from indico.modules.events.models.events import Event from indico.modules.events.timetable.util import get_category_timetable -from indico.modules.events.util import get_base_ical_parameters, serialize_event_for_json_ld from indico.modules.news.util import get_recent_news from indico.modules.users import User from indico.modules.users.models.favorites import favorite_category_table @@ -41,10 +40,9 @@ from indico.util.decorators import classproperty from indico.util.fs import secure_filename from indico.util.i18n import _ -from indico.util.string import to_unicode from indico.web.flask.templating import get_template_module from indico.web.flask.util import send_file, url_for -from indico.web.rh import RH +from indico.web.rh import RH, allow_signed_url from indico.web.util import jsonify_data @@ -58,7 +56,7 @@ def _flat_map(func, list_): - return chain.from_iterable(imap(func, list_)) + return chain.from_iterable(map(func, list_)) class RHCategoryIcon(RHDisplayCategoryBase): @@ -206,7 +204,7 @@ def _process(self): class RHDisplayCategoryEventsBase(RHDisplayCategoryBase): - """Base class for display pages displaying an event list""" + """Base class for display pages displaying an event list.""" _category_query_options = (joinedload('children').load_only('id', 'title', 'protection_mode'), undefer('attachment_count'), undefer('has_events')) @@ -221,109 +219,12 @@ def _process_args(self): RHDisplayCategoryBase._process_args(self) self.now = now_utc(exact=False).astimezone(self.category.display_tzinfo) - def format_event_date(self, event): - day_month = 'dd MMM' - tzinfo = self.category.display_tzinfo - start_dt = event.start_dt.astimezone(tzinfo) - end_dt = event.end_dt.astimezone(tzinfo) - if start_dt.year != end_dt.year: - return '{} - {}'.format(to_unicode(format_date(start_dt, timezone=tzinfo)), - to_unicode(format_date(end_dt, timezone=tzinfo))) - elif (start_dt.month != end_dt.month) or (start_dt.day != end_dt.day): - return '{} - {}'.format(to_unicode(format_date(start_dt, day_month, timezone=tzinfo)), - to_unicode(format_date(end_dt, day_month, timezone=tzinfo))) - else: - return to_unicode(format_date(start_dt, day_month, timezone=tzinfo)) - - def group_by_month(self, events): - def _format_tuple(x): - (year, month), events = x - return {'name': format_date(date(year, month, 1), format='MMMM yyyy'), - 'events': list(events), - 'is_current': year == self.now.year and month == self.now.month} - - def _key(event): - start_dt = event.start_dt.astimezone(self.category.tzinfo) - return start_dt.year, start_dt.month - - months = groupby(events, key=_key) - return map(_format_tuple, months) - - def happening_now(self, event): - return event.start_dt <= self.now < event.end_dt - - def is_recent(self, dt): - return dt > self.now - relativedelta(weeks=1) - class RHDisplayCategory(RHDisplayCategoryEventsBase): """Show the contents of a category (events/subcategories)""" def _process(self): - # Current events, which are always shown by default are events of this month and of the previous month. - # If there are no events in this range, it will include the last and next month containing events. - past_threshold = self.now - relativedelta(months=1, day=1, hour=0, minute=0) - future_threshold = self.now + relativedelta(months=1, day=1, hour=0, minute=0) - hidden_event_ids = {e.id for e in self.category.get_hidden_events(user=session.user)} - next_event_start_dt = (db.session.query(Event.start_dt) - .filter(Event.start_dt >= self.now, Event.category_id == self.category.id, - Event.id.notin_(hidden_event_ids)) - .order_by(Event.start_dt.asc(), Event.id.asc()) - .first() or (None,))[0] - previous_event_start_dt = (db.session.query(Event.start_dt) - .filter(Event.start_dt < self.now, Event.category_id == self.category.id, - Event.id.notin_(hidden_event_ids)) - .order_by(Event.start_dt.desc(), Event.id.desc()) - .first() or (None,))[0] - if next_event_start_dt is not None and next_event_start_dt > future_threshold: - future_threshold = next_event_start_dt + relativedelta(months=1, day=1, hour=0, minute=0) - if previous_event_start_dt is not None and previous_event_start_dt < past_threshold: - past_threshold = previous_event_start_dt.replace(day=1, hour=0, minute=0) - event_query = (Event.query.with_parent(self.category) - .options(*self._event_query_options) - .filter(Event.id.notin_(hidden_event_ids)) - .order_by(Event.start_dt.desc(), Event.id.desc())) - past_event_query = event_query.filter(Event.start_dt < past_threshold) - future_event_query = event_query.filter(Event.start_dt >= future_threshold) - current_event_query = event_query.filter(Event.start_dt >= past_threshold, - Event.start_dt < future_threshold) - json_ld_events = events = current_event_query.filter(Event.start_dt < future_threshold).all() - events_by_month = self.group_by_month(events) - - future_event_count = future_event_query.count() - past_event_count = past_event_query.count() - has_hidden_events = bool(hidden_event_ids) - - if not session.user and future_event_count: - json_ld_events = json_ld_events + future_event_query.all() - - show_future_events = bool(self.category.id in session.get('fetch_future_events_in', set()) or - (session.user and session.user.settings.get('show_future_events', False))) - show_past_events = bool(self.category.id in session.get('fetch_past_events_in', set()) or - (session.user and session.user.settings.get('show_past_events', False))) - - managers = sorted(self.category.get_manager_list(), key=attrgetter('principal_type.name', 'name')) - - threshold_format = '%Y-%m' - params = {'event_count': len(events), - 'events_by_month': events_by_month, - 'format_event_date': self.format_event_date, - 'future_event_count': future_event_count, - 'show_future_events': show_future_events, - 'future_threshold': future_threshold.strftime(threshold_format), - 'happening_now': self.happening_now, - 'is_recent': self.is_recent, - 'managers': managers, - 'past_event_count': past_event_count, - 'show_past_events': show_past_events, - 'past_threshold': past_threshold.strftime(threshold_format), - 'has_hidden_events': has_hidden_events, - 'json_ld': map(serialize_event_for_json_ld, json_ld_events), - 'atom_feed_url': url_for('.export_atom', self.category), - 'atom_feed_title': _('Events of "{}"').format(self.category.title)} - params.update(get_base_ical_parameters(session.user, 'category', - '/export/categ/{0}.ics'.format(self.category.id), {'from': '-31d'})) - + params = get_category_view_params(self.category, self.now) if not self.category.is_root: return WPCategory.render_template('display/category.html', self.category, **params) @@ -334,7 +235,7 @@ def _process(self): class RHEventList(RHDisplayCategoryEventsBase): - """Return the HTML for the event list before/after a specific month""" + """Return the HTML for the event list before/after a specific month.""" def _parse_year_month(self, string): try: @@ -361,15 +262,18 @@ def _process_args(self): self.events = event_query.all() def _process(self): - events_by_month = self.group_by_month(self.events) tpl = get_template_module('categories/display/event_list.html') - html = tpl.event_list_block(events_by_month=events_by_month, format_event_date=self.format_event_date, - is_recent=self.is_recent, happening_now=self.happening_now) + html = tpl.event_list_block(events_by_month=group_by_month(self.events, self.now, self.category.tzinfo), + format_event_date=make_format_event_date_func(self.category), + is_recent=make_is_recent_func(self.now), + happening_now=make_happening_now_func(self.now)) return jsonify_data(flash=False, html=html) class RHShowEventsInCategoryBase(RHDisplayCategoryBase): - """Set whether the events in a category are automatically displayed or not""" + """ + Set whether the events in a category are automatically displayed or not. + """ session_field = '' @@ -389,20 +293,25 @@ def _process_PUT(self): class RHShowFutureEventsInCategory(RHShowEventsInCategoryBase): - """Set whether the past events in a category are automatically displayed or not""" + """ + Set whether the past events in a category are automatically displayed or not. + """ session_field = 'fetch_future_events_in' class RHShowPastEventsInCategory(RHShowEventsInCategoryBase): - """Set whether the past events in a category are automatically displayed or not""" + """ + Set whether the past events in a category are automatically displayed or not. + """ session_field = 'fetch_past_events_in' +@allow_signed_url class RHExportCategoryICAL(RHDisplayCategoryBase): def _process(self): - filename = '{}-category.ics'.format(secure_filename(self.category.title, str(self.category.id))) + filename = f'{secure_filename(self.category.title, str(self.category.id))}-category.ics' buf = serialize_categories_ical([self.category.id], session.user, Event.end_dt >= (now_utc() - timedelta(weeks=4))) return send_file(filename, buf, 'text/calendar') @@ -410,7 +319,7 @@ def _process(self): class RHExportCategoryAtom(RHDisplayCategoryBase): def _process(self): - filename = '{}-category.atom'.format(secure_filename(self.category.title, str(self.category.id))) + filename = f'{secure_filename(self.category.title, str(self.category.id))}-category.atom' buf = serialize_category_atom(self.category, url_for(request.endpoint, self.category, _external=True), session.user, @@ -418,21 +327,8 @@ def _process(self): return send_file(filename, buf, 'application/atom+xml') -class RHXMLExportCategoryInfo(RH): - def _process_args(self): - try: - id_ = int(request.args['id']) - except ValueError: - raise BadRequest('Invalid Category ID') - self.category = Category.get_or_404(id_, is_deleted=False) - - def _process(self): - category_xml_info = XMLCategorySerializer(self.category).serialize_category() - return Response(category_xml_info, mimetype='text/xml') - - class RHCategoryOverview(RHDisplayCategoryBase): - """Display the events for a particular day, week or month""" + """Display the events for a particular day, week or month.""" def _get_timetable(self): return get_category_timetable([self.category.id], self.start_dt, self.end_dt, @@ -571,7 +467,7 @@ def _process_multiday_events(self, info, event): tzinfo = self.category.display_tzinfo # Breaks, contributions and sessions grouped by start_dt. Each EventProxy will return the relevant ones only - timetable_objects = sorted(chain(*info[event.id].values()), key=attrgetter('timetable_entry.start_dt')) + timetable_objects = sorted(chain(*list(info[event.id].values())), key=attrgetter('timetable_entry.start_dt')) timetable_objects_by_date = {x[0]: list(x[1]) for x in groupby(timetable_objects, key=lambda x: x.start_dt.astimezone(tzinfo).date())} @@ -583,7 +479,7 @@ def _process_multiday_events(self, info, event): return [_EventProxy(event, day, tzinfo, timetable_objects_by_date.get(day.date(), [])) for day in event_days] -class _EventProxy(object): +class _EventProxy: def __init__(self, event, date, tzinfo, timetable_objects): start_dt = datetime.combine(date, event.start_dt.astimezone(tzinfo).timetz()) assert date >= event.start_dt @@ -631,6 +527,7 @@ def _process(self): events = self._get_event_data(query) ongoing_events = (Event.query .filter(Event.is_visible_in(self.category.id), + ~Event.is_deleted, Event.start_dt < start, Event.end_dt > end) .options(load_only('id', 'title', 'start_dt', 'end_dt', 'timezone')) diff --git a/indico/modules/categories/controllers/management.py b/indico/modules/categories/controllers/management.py index 6f338269ca7..dd5c5651fca 100644 --- a/indico/modules/categories/controllers/management.py +++ b/indico/modules/categories/controllers/management.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import random from io import BytesIO @@ -33,9 +31,10 @@ from indico.modules.users import User from indico.util.fs import secure_filename from indico.util.i18n import _, ngettext +from indico.util.marshmallow import PrincipalList from indico.util.roles import ImportRoleMembersMixin from indico.util.string import crc32 -from indico.util.user import principal_from_fossil +from indico.web.args import use_kwargs from indico.web.flask.templating import get_template_module from indico.web.flask.util import url_for from indico.web.forms.base import FormDefaults @@ -133,7 +132,7 @@ def _process_POST(self): f = request.files[self.IMAGE_TYPE] try: img = Image.open(f) - except IOError: + except OSError: flash(_('You cannot upload this file as an icon/logo.'), 'error') return jsonify_data(content=None) if img.format.lower() not in {'jpeg', 'png', 'gif'}: @@ -156,13 +155,13 @@ def _process_POST(self): } self._set_image(content, metadata) flash(self.SAVED_FLASH_MSG, 'success') - logger.info("New {} '%s' uploaded by %s (%s)".format(self.IMAGE_TYPE), f.filename, session.user, self.category) + logger.info(f"New {self.IMAGE_TYPE} '%s' uploaded by %s (%s)", f.filename, session.user, self.category) return jsonify_data(content=get_image_data(self.IMAGE_TYPE, self.category)) def _process_DELETE(self): self._set_image(None, None) flash(self.DELETED_FLASH_MSG, 'success') - logger.info("{} of %s deleted by %s".format(self.IMAGE_TYPE.title()), self.category, session.user) + logger.info(f"{self.IMAGE_TYPE.title()} of %s deleted by %s", self.category, session.user) return jsonify_data(content=None) @@ -293,7 +292,7 @@ def _process(self): class RHDeleteSubcategories(RHManageCategoryBase): - """Bulk-delete subcategories""" + """Bulk-delete subcategories.""" def _process_args(self): RHManageCategoryBase._process_args(self) @@ -306,7 +305,7 @@ def _process(self): if 'confirmed' in request.form: for subcategory in self.subcategories: if not subcategory.is_empty: - raise BadRequest('Category "{}" is not empty'.format(subcategory.title)) + raise BadRequest(f'Category "{subcategory.title}" is not empty') delete_category(subcategory) return jsonify_data(flash=False, is_empty=self.category.is_empty) return jsonify_template('categories/management/delete_categories.html', categories=self.subcategories, @@ -318,7 +317,7 @@ class RHMoveSubcategories(RHMoveCategoryBase): def _process_args(self): RHMoveCategoryBase._process_args(self) - subcategory_ids = map(int, request.values.getlist('category_id')) + subcategory_ids = request.values.getlist('category_id', type=int) self.subcategories = (Category.query.with_parent(self.category) .filter(Category.id.in_(subcategory_ids)) .order_by(Category.title) @@ -346,7 +345,7 @@ def _process(self): class RHManageCategorySelectedEventsBase(RHManageCategoryBase): - """Base RH to manage selected events in a category""" + """Base RH to manage selected events in a category.""" def _process_args(self): RHManageCategoryBase._process_args(self) @@ -359,7 +358,7 @@ def _process_args(self): class RHDeleteEvents(RHManageCategorySelectedEventsBase): - """Delete multiple events""" + """Delete multiple events.""" def _process(self): is_submitted = 'confirmed' in request.form @@ -390,9 +389,9 @@ def _process(self): form = SplitCategoryForm(formdata=request.form) if form.validate_on_submit(): self._move_events(self.sel_events, form.first_category.data) - if not form.all_selected.data: + if not form.all_selected.data and form.second_category.data: self._move_events(self.cat_events - self.sel_events, form.second_category.data) - if form.all_selected.data: + if form.all_selected.data or not form.second_category.data: flash(_('Your events have been moved to the category "{}"') .format(form.first_category.data), 'success') else: @@ -425,7 +424,7 @@ def _process(self): class RHCategoryRoles(RHManageCategoryBase): - """Category role management""" + """Category role management.""" def _process(self): return WPCategoryManagement.render_template('management/roles.html', self.category, 'roles', @@ -433,7 +432,7 @@ def _process(self): class RHAddCategoryRole(RHManageCategoryBase): - """Add a new category role""" + """Add a new category role.""" def _process(self): form = CategoryRoleForm(category=self.category, color=self._get_color()) @@ -452,7 +451,7 @@ def _get_color(self): class RHManageCategoryRole(RHManageCategoryBase): - """Base class to manage a specific category role""" + """Base class to manage a specific category role.""" normalize_url_spec = { 'locators': { @@ -466,7 +465,7 @@ def _process_args(self): class RHEditCategoryRole(RHManageCategoryRole): - """Edit a category role""" + """Edit a category role.""" def _process(self): form = CategoryRoleForm(obj=self.role, category=self.category) @@ -479,7 +478,7 @@ def _process(self): class RHDeleteCategoryRole(RHManageCategoryRole): - """Delete a category role""" + """Delete a category role.""" def _process(self): db.session.delete(self.role) @@ -488,7 +487,7 @@ def _process(self): class RHRemoveCategoryRoleMember(RHManageCategoryRole): - """Remove a user from a category role""" + """Remove a user from a category role.""" normalize_url_spec = dict(RHManageCategoryRole.normalize_url_spec, preserved_args={'user_id'}) @@ -504,18 +503,19 @@ def _process(self): class RHAddCategoryRoleMembers(RHManageCategoryRole): - """Add users to a category role""" - - def _process(self): - for data in request.json['users']: - user = principal_from_fossil(data, allow_pending=True, allow_groups=False) - if user not in self.role.members: - self.role.members.add(user) - logger.info('User %r added to role %r by %r', user, self.role, session.user) + """Add users to a category role.""" + + @use_kwargs({ + 'users': PrincipalList(required=True, allow_external_users=True), + }) + def _process(self, users): + for user in users - self.role.members: + self.role.members.add(user) + logger.info('User %r added to role %r by %r', user, self.role, session.user) return jsonify_data(html=_render_role(self.role, collapsed=False)) class RHCategoryRoleMembersImportCSV(ImportRoleMembersMixin, RHManageCategoryRole): - """Add users to a category role from CSV""" + """Add users to a category role from CSV.""" logger = logger diff --git a/indico/modules/categories/controllers/util.py b/indico/modules/categories/controllers/util.py new file mode 100644 index 00000000000..f3d2ab5c07e --- /dev/null +++ b/indico/modules/categories/controllers/util.py @@ -0,0 +1,131 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from datetime import date +from itertools import groupby +from operator import attrgetter + +from dateutil.relativedelta import relativedelta +from flask import session + +from indico.core.db import db +from indico.modules.events.models.events import Event +from indico.modules.events.util import serialize_event_for_json_ld +from indico.util.date_time import format_date +from indico.util.i18n import _, pgettext +from indico.web.flask.util import url_for + + +def group_by_month(events, now, tzinfo): + date_format = pgettext('babel date format for category event list headers (month/year only)', 'MMMM yyyy') + + def _format_tuple(x): + (year, month), events = x + return {'name': format_date(date(year, month, 1), format=date_format), + 'events': list(events), + 'is_current': year == now.year and month == now.month} + + def _key(event): + start_dt = event.start_dt.astimezone(tzinfo) + return start_dt.year, start_dt.month + + months = groupby(events, key=_key) + return list(map(_format_tuple, months)) + + +def make_format_event_date_func(category): + day_month = pgettext('babel date format for category event list event entries (day/month only)', 'dd MMM') + + def fn(event): + tzinfo = category.display_tzinfo + start_dt = event.start_dt.astimezone(tzinfo) + end_dt = event.end_dt.astimezone(tzinfo) + if start_dt.year != end_dt.year: + return '{} - {}'.format(format_date(start_dt, timezone=tzinfo), + format_date(end_dt, timezone=tzinfo)) + elif (start_dt.month != end_dt.month) or (start_dt.day != end_dt.day): + return '{} - {}'.format(format_date(start_dt, day_month, timezone=tzinfo), + format_date(end_dt, day_month, timezone=tzinfo)) + else: + return format_date(start_dt, day_month, timezone=tzinfo) + + return fn + + +def make_happening_now_func(now): + return lambda event: event.start_dt <= now < event.end_dt + + +def make_is_recent_func(now): + return lambda dt: dt > now - relativedelta(weeks=1) + + +def get_category_view_params(category, now): + from .display import RHDisplayCategoryEventsBase + + # Current events, which are always shown by default are events of this month and of the previous month. + # If there are no events in this range, it will include the last and next month containing events. + past_threshold = now - relativedelta(months=1, day=1, hour=0, minute=0) + future_threshold = now + relativedelta(months=1, day=1, hour=0, minute=0) + hidden_event_ids = {e.id for e in category.get_hidden_events(user=session.user)} + next_event_start_dt = (db.session.query(Event.start_dt) + .filter(Event.start_dt >= now, Event.category_id == category.id, + Event.id.notin_(hidden_event_ids)) + .order_by(Event.start_dt.asc(), Event.id.asc()) + .first() or (None,))[0] + previous_event_start_dt = (db.session.query(Event.start_dt) + .filter(Event.start_dt < now, Event.category_id == category.id, + Event.id.notin_(hidden_event_ids)) + .order_by(Event.start_dt.desc(), Event.id.desc()) + .first() or (None,))[0] + if next_event_start_dt is not None and next_event_start_dt > future_threshold: + future_threshold = next_event_start_dt + relativedelta(months=1, day=1, hour=0, minute=0) + if previous_event_start_dt is not None and previous_event_start_dt < past_threshold: + past_threshold = previous_event_start_dt.replace(day=1, hour=0, minute=0) + event_query = (Event.query.with_parent(category) + .options(*RHDisplayCategoryEventsBase._event_query_options) + .filter(Event.id.notin_(hidden_event_ids)) + .order_by(Event.start_dt.desc(), Event.id.desc())) + past_event_query = event_query.filter(Event.start_dt < past_threshold) + future_event_query = event_query.filter(Event.start_dt >= future_threshold) + current_event_query = event_query.filter(Event.start_dt >= past_threshold, + Event.start_dt < future_threshold) + json_ld_events = events = current_event_query.filter(Event.start_dt < future_threshold).all() + + future_event_count = future_event_query.count() + past_event_count = past_event_query.count() + has_hidden_events = bool(hidden_event_ids) + + if not session.user and future_event_count: + json_ld_events = json_ld_events + future_event_query.all() + + show_future_events = bool(category.id in session.get('fetch_future_events_in', set()) or + (session.user and session.user.settings.get('show_future_events', False))) + show_past_events = bool(category.id in session.get('fetch_past_events_in', set()) or + (session.user and session.user.settings.get('show_past_events', False))) + + managers = sorted(category.get_manager_list(), key=attrgetter('principal_type.name', 'name')) + + threshold_format = '%Y-%m' + return { + 'event_count': len(events), + 'events_by_month': group_by_month(events, now, category.tzinfo), + 'format_event_date': make_format_event_date_func(category), + 'future_event_count': future_event_count, + 'show_future_events': show_future_events, + 'future_threshold': future_threshold.strftime(threshold_format), + 'happening_now': make_happening_now_func(now), + 'is_recent': make_is_recent_func(now), + 'managers': managers, + 'past_event_count': past_event_count, + 'show_past_events': show_past_events, + 'past_threshold': past_threshold.strftime(threshold_format), + 'has_hidden_events': has_hidden_events, + 'json_ld': list(map(serialize_event_for_json_ld, json_ld_events)), + 'atom_feed_url': url_for('.export_atom', category), + 'atom_feed_title': _('Events of "{}"').format(category.title) + } diff --git a/indico/modules/categories/fields.py b/indico/modules/categories/fields.py index f989a7ee591..3b67870777e 100644 --- a/indico/modules/categories/fields.py +++ b/indico/modules/categories/fields.py @@ -1,27 +1,20 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import json from wtforms.fields.simple import HiddenField -from indico.util.i18n import _ from indico.web.forms.widgets import JinjaWidget class CategoryField(HiddenField): """WTForms field that lets you select a category. - :param allow_events: Whether to allow selecting a category that - contains events. - :param allow_subcats: Whether to allow selecting a category that - contains subcategories. :param require_event_creation_rights: Whether to allow selecting only categories where the user can create events. @@ -31,27 +24,15 @@ class CategoryField(HiddenField): def __init__(self, *args, **kwargs): self.navigator_category_id = 0 - self.allow_events = kwargs.pop('allow_events', True) - self.allow_subcats = kwargs.pop('allow_subcats', True) self.require_event_creation_rights = kwargs.pop('require_event_creation_rights', False) - super(CategoryField, self).__init__(*args, **kwargs) - - def pre_validate(self, form): - if self.data: - self._validate(self.data) + super().__init__(*args, **kwargs) def process_data(self, value): if not value: self.data = None return - try: - self._validate(value) - except ValueError: - self.data = None - self.navigator_category_id = value.id - else: - self.data = value - self.navigator_category_id = value.id + self.data = value + self.navigator_category_id = value.id def process_formdata(self, valuelist): from indico.modules.categories import Category @@ -63,12 +44,6 @@ def process_formdata(self, valuelist): else: self.data = Category.get(category_id, is_deleted=False) - def _validate(self, category): - if not self.allow_events and category.has_only_events: - raise ValueError(_("Categories containing only events are not allowed.")) - if not self.allow_subcats and category.children: - raise ValueError(_("Categories containing subcategories are not allowed.")) - def _value(self): return {'id': self.data.id, 'title': self.data.title} if self.data else {} diff --git a/indico/modules/categories/forms.py b/indico/modules/categories/forms.py index 3c4e74ab3f5..17278f14430 100644 --- a/indico/modules/categories/forms.py +++ b/indico/modules/categories/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import partial from flask import request @@ -22,7 +20,7 @@ from indico.modules.events.models.events import EventType from indico.modules.networks import IPNetworkGroup from indico.util.i18n import _ -from indico.util.user import principal_from_fossil +from indico.util.user import principal_from_identifier from indico.web.forms.base import IndicoForm from indico.web.forms.colors import get_role_colors from indico.web.forms.fields import (EditableFileField, EmailListField, HiddenFieldList, IndicoEnumSelectField, @@ -97,7 +95,7 @@ class CategoryProtectionForm(IndicoForm): def __init__(self, *args, **kwargs): self.protected_object = self.category = kwargs.pop('category') - super(CategoryProtectionForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self._init_visibility() def _init_visibility(self): @@ -111,8 +109,11 @@ def _init_visibility(self): def validate_permissions(self, field): for principal_fossil, permissions in field.data: - principal = principal_from_fossil(principal_fossil, allow_networks=True, allow_pending=True, - category=self.category) + principal = principal_from_identifier(principal_fossil['identifier'], + allow_groups=True, + allow_networks=True, + allow_category_roles=True, + category_id=self.category.id) if isinstance(principal, IPNetworkGroup) and set(permissions) - {READ_ACCESS_PERMISSION}: msg = _('IP networks cannot have management permissions: {}').format(principal.name) raise ValidationError(msg) @@ -122,7 +123,7 @@ def validate_permissions(self, field): class CreateCategoryForm(IndicoForm): - """Form to create a new Category""" + """Form to create a new Category.""" title = StringField(_("Title"), [DataRequired()]) description = IndicoMarkdownField(_("Description")) @@ -132,15 +133,16 @@ class SplitCategoryForm(IndicoForm): first_category = StringField(_('Category name #1'), [DataRequired()], description=_('Selected events will be moved into a new sub-category with this ' 'title.')) - second_category = StringField(_('Category name #2'), [DataRequired()], + second_category = StringField(_('Category name #2'), description=_('Events that were not selected will be moved into a new sub-category ' - 'with this title.')) + 'with this title. If omitted, those events will remain in the current ' + 'category.')) event_id = HiddenFieldList() all_selected = BooleanField(widget=HiddenCheckbox()) submitted = HiddenField() def __init__(self, *args, **kwargs): - super(SplitCategoryForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) if self.all_selected.data: self.event_id.data = [] self.first_category.label.text = _('Category name') @@ -148,7 +150,7 @@ def __init__(self, *args, **kwargs): del self.second_category def is_submitted(self): - return super(SplitCategoryForm, self).is_submitted() and 'submitted' in request.form + return super().is_submitted() and 'submitted' in request.form class UpcomingEventsForm(IndicoForm): @@ -196,7 +198,7 @@ class CategoryRoleForm(IndicoForm): def __init__(self, *args, **kwargs): self.role = kwargs.get('obj') self.category = kwargs.pop('category') - super(CategoryRoleForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def validate_code(self, field): query = CategoryRole.query.with_parent(self.category).filter_by(code=field.data) diff --git a/indico/modules/categories/legacy.py b/indico/modules/categories/legacy.py index 956803a2974..1c410eea0f1 100644 --- a/indico/modules/categories/legacy.py +++ b/indico/modules/categories/legacy.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from lxml import etree -class XMLCategorySerializer(object): +class XMLCategorySerializer: def __init__(self, category): self.category = category diff --git a/indico/modules/categories/models/__init__.py b/indico/modules/categories/models/__init__.py index bd54f91c245..9badb30d622 100644 --- a/indico/modules/categories/models/__init__.py +++ b/indico/modules/categories/models/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import textwrap from sqlalchemy import DDL diff --git a/indico/modules/categories/models/categories.py b/indico/modules/categories/models/categories.py index 3dfcb228367..67f41d16397 100644 --- a/indico/modules/categories/models/categories.py +++ b/indico/modules/categories/models/categories.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import pytz from sqlalchemy import DDL, orm from sqlalchemy.dialects.postgresql import ARRAY, JSONB, array @@ -27,16 +25,16 @@ from indico.core.db.sqlalchemy.util.models import auto_table_args from indico.util.date_time import get_display_tz from indico.util.decorators import strict_classproperty +from indico.util.enum import RichIntEnum from indico.util.i18n import _ from indico.util.locators import locator_property -from indico.util.string import MarkdownText, format_repr, return_ascii, text_to_repr -from indico.util.struct.enum import RichIntEnum +from indico.util.string import MarkdownText, format_repr, text_to_repr from indico.web.flask.util import url_for def _get_next_position(context): parent_id = context.current_parameters['parent_id'] - res = db.session.query(db.func.max(Category.position)).filter_by(parent_id=parent_id).one() + res = db.session.query(db.func.max(Category.position)).filter(Category.parent_id == parent_id).one() return (res[0] or 0) + 1 @@ -57,7 +55,7 @@ class EventMessageMode(RichIntEnum): class Category(SearchableTitleMixin, DescriptionMixin, ProtectionManagersMixin, AttachedItemsMixin, db.Model): - """An Indico category""" + """An Indico category.""" __tablename__ = 'categories' disallowed_protection_modes = frozenset() @@ -74,7 +72,7 @@ def __auto_table_args(cls): db.CheckConstraint("(logo IS NULL) = (logo_metadata::text = 'null')", 'valid_logo'), db.CheckConstraint("(parent_id IS NULL) = (id = 0)", 'valid_parent'), db.CheckConstraint("(id != 0) OR NOT is_deleted", 'root_not_deleted'), - db.CheckConstraint("(id != 0) OR (protection_mode != {})".format(ProtectionMode.inheriting), + db.CheckConstraint(f"(id != 0) OR (protection_mode != {ProtectionMode.inheriting})", 'root_not_inheriting'), db.CheckConstraint('visibility IS NULL OR visibility > 0', 'valid_visibility'), {'schema': 'categories'}) @@ -234,7 +232,6 @@ def event_message(self, value): def event_message(cls): return cls._event_message - @return_ascii def __repr__(self): return format_repr(self, 'id', is_deleted=False, _text=text_to_repr(self.title, max_length=75)) @@ -248,17 +245,13 @@ def locator(self): @classmethod def get_root(cls): - """Get the root category""" + """Get the root category.""" return cls.query.filter(cls.is_root).one() @property def url(self): return url_for('categories.display', self) - @property - def has_only_events(self): - return self.has_events and not self.children - @hybrid_property def is_root(self): return self.parent_id is None @@ -289,7 +282,7 @@ def tzinfo(self): @property def display_tzinfo(self): - """The tzinfo of the category or the one specified by the user""" + """The tzinfo of the category or the one specified by the user.""" return get_display_tz(self, as_timezone=True) def can_create_events(self, user): @@ -459,7 +452,10 @@ def visibility_horizon_query(self): @property def own_visibility_horizon(self): - """Get the highest category this one would like to be visible from (configured visibility).""" + """ + Get the highest category this one would like to be visible + from (configured visibility). + """ if self.visibility is None: return Category.get_root() else: @@ -467,7 +463,10 @@ def own_visibility_horizon(self): @property def real_visibility_horizon(self): - """Get the highest category this one is actually visible from (as limited by categories above).""" + """ + Get the highest category this one is actually visible + from (as limited by categories above). + """ horizon_id, final_visibility = self.visibility_horizon_query.one() if final_visibility is not None and final_visibility < 0: return None # Category is invisible @@ -540,7 +539,7 @@ def _mappers_configured(): # (public/protected) of the category, even if it's inheriting it from its # parent category cte = Category.get_protection_cte() - query = select([cte.c.protection_mode]).where(cte.c.id == Category.id).correlate_except(cte) + query = select([cte.c.protection_mode]).where(cte.c.id == Category.id).correlate_except(cte).scalar_subquery() Category.effective_protection_mode = column_property(query, deferred=True, expire_on_flush=False) # Category.effective_icon_data -- the effective icon data of the category, @@ -549,14 +548,16 @@ def _mappers_configured(): query = (select([db.func.json_build_object('source_id', cte.c.source_id, 'metadata', cte.c.icon_metadata)]) .where(cte.c.id == Category.id) - .correlate_except(cte)) + .correlate_except(cte) + .scalar_subquery()) Category.effective_icon_data = column_property(query, deferred=True) # Category.event_count -- the number of events in the category itself, # excluding deleted events query = (select([db.func.count(Event.id)]) .where((Event.category_id == Category.id) & ~Event.is_deleted) - .correlate_except(Event)) + .correlate_except(Event) + .scalar_subquery()) Category.event_count = column_property(query, deferred=True) # Category.has_events -- whether the category itself contains any @@ -569,14 +570,14 @@ def _mappers_configured(): # Category.chain_titles -- a list of the titles in the parent chain, # starting with the root category down to the current category. cte = Category.get_tree_cte('title') - query = select([cte.c.path]).where(cte.c.id == Category.id).correlate_except(cte) + query = select([cte.c.path]).where(cte.c.id == Category.id).correlate_except(cte).scalar_subquery() Category.chain_titles = column_property(query, deferred=True) # Category.chain -- a list of the ids and titles in the parent # chain, starting with the root category down to the current # category. Each chain entry is a dict containing 'id' and `title`. cte = Category.get_tree_cte(lambda cat: db.func.json_build_object('id', cat.id, 'title', cat.title)) - query = select([cte.c.path]).where(cte.c.id == Category.id).correlate_except(cte) + query = select([cte.c.path]).where(cte.c.id == Category.id).correlate_except(cte).scalar_subquery() Category.chain = column_property(query, deferred=True) # Category.deep_events_count -- the number of events in the category @@ -586,7 +587,7 @@ def _mappers_configured(): cte.c.path.contains(array([Category.id])), ~cte.c.is_deleted, ~Event.is_deleted) - query = select([db.func.count()]).where(crit).correlate_except(Event) + query = select([db.func.count()]).where(crit).correlate_except(Event).scalar_subquery() Category.deep_events_count = column_property(query, deferred=True) # Category.deep_children_count -- the number of subcategories in the @@ -594,7 +595,7 @@ def _mappers_configured(): cte = Category.get_tree_cte() crit = db.and_(cte.c.path.contains(array([Category.id])), cte.c.id != Category.id, ~cte.c.is_deleted) - query = select([db.func.count()]).where(crit).correlate_except(cte) + query = select([db.func.count()]).where(crit).correlate_except(cte).scalar_subquery() Category.deep_children_count = column_property(query, deferred=True) diff --git a/indico/modules/categories/models/categories_test.py b/indico/modules/categories/models/categories_test.py index 08ea25f41ae..7b2d3dc235c 100644 --- a/indico/modules/categories/models/categories_test.py +++ b/indico/modules/categories/models/categories_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -39,7 +39,7 @@ def test_can_create_events_no_user(dummy_category): def test_effective_protection_mode(db): def _cat(id_, protection_mode=ProtectionMode.inheriting, children=None): - return Category(id=id_, title='cat-{}'.format(id_), protection_mode=protection_mode, children=children or []) + return Category(id=id_, title=f'cat-{id_}', protection_mode=protection_mode, children=children or []) root = Category.get_root() root.protection_mode = ProtectionMode.protected root.children = [ diff --git a/indico/modules/categories/models/legacy_mapping.py b/indico/modules/categories/models/legacy_mapping.py index 2246204f320..b1f5b654f9d 100644 --- a/indico/modules/categories/models/legacy_mapping.py +++ b/indico/modules/categories/models/legacy_mapping.py @@ -1,18 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db -from indico.util.string import return_ascii class LegacyCategoryMapping(db.Model): - """Legacy category ID mapping + """Legacy category ID mapping. Legacy categories have non-numeric IDs which are not supported by any new code. This mapping maps them to proper integer IDs to @@ -45,6 +42,5 @@ class LegacyCategoryMapping(db.Model): ) ) - @return_ascii def __repr__(self): - return ''.format(self.legacy_category_id, self.category_id) + return f'' diff --git a/indico/modules/categories/models/principals.py b/indico/modules/categories/models/principals.py index d7f7a986289..13368791aaa 100644 --- a/indico/modules/categories/models/principals.py +++ b/indico/modules/categories/models/principals.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db from indico.core.db.sqlalchemy.principals import PrincipalPermissionsMixin from indico.core.db.sqlalchemy.util.models import auto_table_args -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class CategoryPrincipal(PrincipalPermissionsMixin, db.Model): @@ -43,6 +41,5 @@ def __table_args__(cls): # relationship backrefs: # - category (Category.acl_entries) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'category_id', 'principal', read_access=False, full_access=False, permissions=[]) diff --git a/indico/modules/categories/models/roles.py b/indico/modules/categories/models/roles.py index cdebc0902df..a9b5877ad76 100644 --- a/indico/modules/categories/models/roles.py +++ b/indico/modules/categories/models/roles.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy.principals import PrincipalType from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class CategoryRole(db.Model): @@ -19,12 +17,6 @@ class CategoryRole(db.Model): db.Index(None, 'category_id', 'code', unique=True), {'schema': 'categories'}) - is_group = False - is_event_role = False - is_registration_form = False - is_category_role = True - is_single_person = True - is_network = False principal_order = 2 principal_type = PrincipalType.category_role @@ -81,7 +73,6 @@ class CategoryRole(db.Model): def __contains__(self, user): return user is not None and self in user.category_roles - @return_ascii def __repr__(self): return format_repr(self, 'id', 'code', _text=self.name) @@ -91,7 +82,7 @@ def locator(self): @property def identifier(self): - return 'CategoryRole:{}'.format(self.id) + return f'CategoryRole:{self.id}' @property def css(self): diff --git a/indico/modules/categories/models/settings.py b/indico/modules/categories/models/settings.py index a816c973ff3..c331cceab01 100644 --- a/indico/modules/categories/models/settings.py +++ b/indico/modules/categories/models/settings.py @@ -1,19 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db from indico.core.db.sqlalchemy.util.models import auto_table_args from indico.core.settings.models.base import JSONSettingsBase from indico.util.decorators import strict_classproperty -from indico.util.string import return_ascii class CategorySetting(JSONSettingsBase, db.Model): @@ -45,6 +42,5 @@ def __table_args__(cls): ) ) - @return_ascii def __repr__(self): - return ''.format(self.category_id, self.module, self.name, self.value) + return f'' diff --git a/indico/modules/categories/operations.py b/indico/modules/categories/operations.py index 593d0050c0e..6666dbb12ab 100644 --- a/indico/modules/categories/operations.py +++ b/indico/modules/categories/operations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals diff --git a/indico/modules/categories/serialize.py b/indico/modules/categories/serialize.py index 5b2c9805351..838e7af5611 100644 --- a/indico/modules/categories/serialize.py +++ b/indico/modules/categories/serialize.py @@ -1,32 +1,24 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from io import BytesIO -from itertools import ifilter -import icalendar as ical from feedgen.feed import FeedGenerator from flask import session -from lxml import html -from lxml.etree import ParserError from sqlalchemy.orm import joinedload, load_only, subqueryload, undefer -from werkzeug.urls import url_parse -from indico.core.config import config from indico.modules.categories import Category from indico.modules.events import Event -from indico.util.date_time import now_utc +from indico.modules.events.ical import events_to_ical from indico.util.string import sanitize_html def serialize_categories_ical(category_ids, user, event_filter=True, event_filter_fn=None, update_query=None): - """Export the events in a category to iCal + """Export the events in a category to iCal. :param category_ids: Category IDs to export :param user: The user who needs to be able to access the events @@ -56,7 +48,7 @@ def serialize_categories_ical(category_ids, user, event_filter=True, event_filte query = update_query(query) it = iter(query) if event_filter_fn: - it = ifilter(event_filter_fn, it) + it = filter(event_filter_fn, it) events = list(it) # make sure the parent categories are in sqlalchemy's identity cache. # this avoids query spam from `protection_parent` lookups @@ -64,46 +56,12 @@ def serialize_categories_ical(category_ids, user, event_filter=True, event_filte .options(load_only('id', 'parent_id', 'protection_mode'), joinedload('acl_entries')) .all()) - cal = ical.Calendar() - cal.add('version', '2.0') - cal.add('prodid', '-//CERN//INDICO//EN') - now = now_utc(False) - for event in events: - if not event.can_access(user): - continue - location = ('{} ({})'.format(event.room_name, event.venue_name) - if event.venue_name and event.room_name - else (event.venue_name or event.room_name)) - cal_event = ical.Event() - cal_event.add('uid', u'indico-event-{}@{}'.format(event.id, url_parse(config.BASE_URL).host)) - cal_event.add('dtstamp', now) - cal_event.add('dtstart', event.start_dt) - cal_event.add('dtend', event.end_dt) - cal_event.add('url', event.external_url) - cal_event.add('summary', event.title) - cal_event.add('location', location) - description = [] - if event.person_links: - speakers = [u'{} ({})'.format(x.full_name, x.affiliation) if x.affiliation else x.full_name - for x in event.person_links] - description.append(u'Speakers: {}'.format(u', '.join(speakers))) - - if event.description: - desc_text = unicode(event.description) or u'

' # get rid of RichMarkup - try: - description.append(unicode(html.fromstring(desc_text).text_content())) - except ParserError: - # this happens e.g. if desc_text contains only a html comment - pass - description.append(event.external_url) - cal_event.add('description', u'\n'.join(description)) - cal.add_component(cal_event) - return BytesIO(cal.to_ical()) + return BytesIO(events_to_ical(events, user)) def serialize_category_atom(category, url, user, event_filter): - """Export the events in a category to Atom + """Export the events in a category to Atom. :param category: The category to export :param url: The URL of the feed @@ -124,14 +82,14 @@ def serialize_category_atom(category, url, user, event_filter): feed = FeedGenerator() feed.id(url) - feed.title('Indico Feed [{}]'.format(category.title)) + feed.title(f'Indico Feed [{category.title}]') feed.link(href=url, rel='self') for event in events: entry = feed.add_entry(order='append') entry.id(event.external_url) entry.title(event.title) - entry.summary(sanitize_html(unicode(event.description)) or None, type='html') + entry.summary(sanitize_html(str(event.description)) or None, type='html') entry.link(href=event.external_url) entry.updated(event.start_dt) return BytesIO(feed.atom_str(pretty=True)) diff --git a/indico/modules/categories/settings.py b/indico/modules/categories/settings.py index ba287092155..6ce5b2165c3 100644 --- a/indico/modules/categories/settings.py +++ b/indico/modules/categories/settings.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import wraps from indico.core.settings import SettingsProxyBase @@ -31,7 +29,7 @@ class CategorySettingsProxy(SettingsProxyBase): @property def query(self): """Return a query object filtering by the proxy's module.""" - return CategorySetting.find(module=self.module) + return CategorySetting.query.filter_by(module=self.module) @_category_or_id def get_all(self, category, no_defaults=False): @@ -74,7 +72,7 @@ def set_multi(self, category, items): :param category: Category (or its ID) :param items: Dict containing the new settings """ - items = {k: self._convert_from_python(k, v) for k, v in items.iteritems()} + items = {k: self._convert_from_python(k, v) for k, v in items.items()} CategorySetting.set_multi(self.module, items, category_id=category) self._flush_cache() diff --git a/indico/modules/categories/tasks.py b/indico/modules/categories/tasks.py index 049b07d7324..5ea55ed10b7 100644 --- a/indico/modules/categories/tasks.py +++ b/indico/modules/categories/tasks.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import timedelta from celery.schedules import crontab @@ -36,7 +34,7 @@ def category_suggestions(): for user in users: existing = {x.category: x for x in user.suggested_categories} related = set(get_related_categories(user, detailed=False)) - for category, score in get_category_scores(user).iteritems(): + for category, score in get_category_scores(user).items(): if score < SUGGESTION_MIN_SCORE: continue if (category in related or category.is_deleted or category.suggestions_disabled or @@ -55,7 +53,7 @@ def category_cleanup(): janitor_user = User.get_system_user() logger.debug("Checking whether any categories should be cleaned up") - for categ_id, days in config.CATEGORY_CLEANUP.iteritems(): + for categ_id, days in config.CATEGORY_CLEANUP.items(): try: category = Category.get(int(categ_id), is_deleted=False) except KeyError: diff --git a/indico/modules/categories/templates/category_export_ical.html b/indico/modules/categories/templates/category_export_ical.html deleted file mode 100644 index 4cb7d0f021e..00000000000 --- a/indico/modules/categories/templates/category_export_ical.html +++ /dev/null @@ -1,17 +0,0 @@ -{% extends '_ical_export.html' %} - -{% block download_text %} - {% trans %}Download current category:{% endtrans %} -{% endblock %} - -{% block javascript %} - {{ super() }} - - -{% endblock %} diff --git a/indico/modules/categories/templates/display/base.html b/indico/modules/categories/templates/display/base.html index e6c64e8b077..e32c068accd 100644 --- a/indico/modules/categories/templates/display/base.html +++ b/indico/modules/categories/templates/display/base.html @@ -4,6 +4,8 @@

{% if category.is_root or category.attachment_count or category.can_manage(session.user) %} {% set category_title_classes = 'sidebar-padding' %} + {% else %} + {% set category_title_classes = '' %} {% endif %}

{% block title %} @@ -19,7 +21,7 @@

{% endblock %}

{% block cat_toolbar %} -
+
{% block cat_create_event %} {{ create_event_button(category, classes="highlight", text=_("Create event"), with_tooltip=false) }} @@ -42,13 +44,7 @@

{% endif %} {% endblock %} {% block cat_export %} - - {% with item=category, ics_url=url_for('categories.export_ical', category) -%} - {% include 'categories/category_export_ical.html' %} - {%- endwith %} + {% endblock %} {% block cat_view_dropdown %} } }); }); - $('.js-export-ical').on('click', function(evt) { - evt.preventDefault(); - $(this).trigger('menu_select'); - }); setupCategoryDisplay(); + {% if not category.is_root %} + + {% endif %} {% endblock %} {% block side_menu %} diff --git a/indico/modules/categories/templates/management/content.html b/indico/modules/categories/templates/management/content.html index d5d87e78691..594a33ca466 100644 --- a/indico/modules/categories/templates/management/content.html +++ b/indico/modules/categories/templates/management/content.html @@ -1,17 +1,9 @@ {% extends 'categories/management/base.html' %} {% from 'categories/management/_events_list.html' import render_events_list %} +{% from 'categories/management/_create_category_button.html' import create_category_button %} {% from 'events/management/_create_event_button.html' import create_event_button %} -{% macro _create_category_button(classes='') %} - - {% trans %}New category{% endtrans %} - -{% endmacro %} - {% macro subcategory_row(subcategory) %}

{% endmacro %} -{% macro subcategories_table(subcategories) %} +{% macro subcategories_table(subcategories, has_events=false) %}
- - - {{ _create_category_button(classes='highlight icon-plus') }} +
+ + +
+
+ {{ create_category_button(category, classes='highlight icon-plus') }} + {% if not has_events %} + {{ create_event_button(category, text=_("Create event")) }} + {% endif %} +
diff --git a/indico/modules/events/editing/client/js/management/editable_type/EditableList.module.scss b/indico/modules/events/editing/client/js/management/editable_type/EditableList.module.scss index 67e09becf0c..f9b7166bd01 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/EditableList.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/EditableList.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.jsx b/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.jsx index d16de22c535..3d549f14e1d 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.jsx @@ -1,37 +1,40 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. /* global ajaxDialog:false */ +import anonymousTeamURL from 'indico-url:event_editing.api_anonymous_team'; +import enableEditingURL from 'indico-url:event_editing.api_editing_enabled'; +import selfAssignURL from 'indico-url:event_editing.api_self_assign_enabled'; +import enableSubmissionURL from 'indico-url:event_editing.api_submission_enabled'; +import contactEditingTeamURL from 'indico-url:event_editing.contact_team'; import dashboardURL from 'indico-url:event_editing.dashboard'; import manageEditableTypeListURL from 'indico-url:event_editing.manage_editable_type_list'; import manageFileTypesURL from 'indico-url:event_editing.manage_file_types'; import manageReviewConditionsURL from 'indico-url:event_editing.manage_review_conditions'; -import selfAssignURL from 'indico-url:event_editing.api_self_assign_enabled'; -import enableSubmissionURL from 'indico-url:event_editing.api_submission_enabled'; -import enableEditingURL from 'indico-url:event_editing.api_editing_enabled'; -import contactEditingTeamURL from 'indico-url:event_editing.contact_team'; import React, {useState} from 'react'; import {useParams, Link} from 'react-router-dom'; import {Checkbox, Loader} from 'semantic-ui-react'; -import {Translate} from 'indico/react/i18n'; import {ManagementPageSubTitle, ManagementPageBackButton} from 'indico/react/components'; -import {useNumericParam} from 'indico/react/util/routing'; import {useTogglableValue} from 'indico/react/hooks'; +import {Translate} from 'indico/react/i18n'; +import {useNumericParam} from 'indico/react/util/routing'; + import {EditableTypeTitles, GetNextEditableTitles} from '../../models'; import Section from '../Section'; -import TeamManager from './TeamManager'; + import NextEditable from './NextEditable'; +import TeamManager from './TeamManager'; import './EditableTypeDashboard.module.scss'; export default function EditableTypeDashboard() { - const eventId = useNumericParam('confId'); + const eventId = useNumericParam('event_id'); const {type} = useParams(); const [editorManagerVisible, setEditorManagerVisible] = useState(false); const [selfAssignModalVisible, setSelfAssignModalVisible] = useState(false); @@ -41,19 +44,26 @@ export default function EditableTypeDashboard() { toggleSelfAssign, selfAssignLoading, selfAssignSaving, - ] = useTogglableValue(selfAssignURL({confId: eventId, type})); + ] = useTogglableValue(selfAssignURL({event_id: eventId, type})); + + const [ + anonymousTeamEnabled, + toggleAnonymousTeam, + anonymousTeamLoading, + anonymousTeamSaving, + ] = useTogglableValue(anonymousTeamURL({event_id: eventId, type})); const [submissionEnabled, toggleSubmission, submissionLoading] = useTogglableValue( - enableSubmissionURL({confId: eventId, type}) + enableSubmissionURL({event_id: eventId, type}) ); const [editingEnabled, toggleEditing, editingLoading] = useTogglableValue( - enableEditingURL({confId: eventId, type}) + enableEditingURL({event_id: eventId, type}) ); const contactEditingTeam = () => { ajaxDialog({ - url: contactEditingTeamURL({confId: eventId, type}), + url: contactEditingTeamURL({event_id: eventId, type}), title: Translate.string('Send emails to the editing team'), }); }; @@ -67,8 +77,8 @@ export default function EditableTypeDashboard() { return ( <> - - {selfAssignLoading || submissionLoading || editingLoading ? ( + + {selfAssignLoading || anonymousTeamLoading || submissionLoading || editingLoading ? ( ) : ( <> @@ -108,7 +118,7 @@ export default function EditableTypeDashboard() { > Configure @@ -120,7 +130,7 @@ export default function EditableTypeDashboard() { > Configure @@ -130,6 +140,13 @@ export default function EditableTypeDashboard() { label={Translate.string('Editing team')} description={Translate.string('Configure editing team')} > + Contact @@ -154,7 +171,7 @@ export default function EditableTypeDashboard() { /> List diff --git a/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.module.scss b/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.module.scss index bbdb19aa7e1..dcbc0af9a55 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/EditableTypeDashboard.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/EditableTypeSubPageNav.jsx b/indico/modules/events/editing/client/js/management/editable_type/EditableTypeSubPageNav.jsx index 76f3e5ed60d..651cfb20c94 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/EditableTypeSubPageNav.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/EditableTypeSubPageNav.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -7,21 +7,22 @@ import editableTypeURL from 'indico-url:event_editing.manage_editable_type'; -import React from 'react'; import PropTypes from 'prop-types'; +import React from 'react'; import {useParams} from 'react-router-dom'; -import {useNumericParam} from 'indico/react/util/routing'; import { ManagementPageBackButton, ManagementPageTitle, ManagementPageSubTitle, } from 'indico/react/components'; import {Translate} from 'indico/react/i18n'; +import {useNumericParam} from 'indico/react/util/routing'; + import {EditableTypeTitles} from '../../models'; export default function EditableTypeSubPageNav({title}) { - const eventId = useNumericParam('confId'); + const eventId = useNumericParam('event_id'); const {type} = useParams(); return ( <> @@ -29,7 +30,7 @@ export default function EditableTypeSubPageNav({title}) { title={Translate.string('Editing ({type})', {type: EditableTypeTitles[type]})} /> - + ); } diff --git a/indico/modules/events/editing/client/js/management/editable_type/NextEditable.jsx b/indico/modules/events/editing/client/js/management/editable_type/NextEditable.jsx index 4f5152e4429..dc476ce7844 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/NextEditable.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/NextEditable.jsx @@ -1,31 +1,33 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import editableListURL from 'indico-url:event_editing.api_filter_editables_by_filetypes'; import assignMyselfURL from 'indico-url:event_editing.api_assign_editable_self'; import fileTypesURL from 'indico-url:event_editing.api_file_types'; +import editableListURL from 'indico-url:event_editing.api_filter_editables_by_filetypes'; import _ from 'lodash'; +import PropTypes from 'prop-types'; import React, {useState, useEffect} from 'react'; import {useHistory} from 'react-router-dom'; -import PropTypes from 'prop-types'; import {Button, Loader, Modal, Table, Checkbox, Dimmer} from 'semantic-ui-react'; -import {camelizeKeys} from 'indico/utils/case'; + +import {useIndicoAxios} from 'indico/react/hooks'; import {Translate} from 'indico/react/i18n'; import {indicoAxios, handleAxiosError} from 'indico/utils/axios'; -import {useIndicoAxios} from 'indico/react/hooks'; -import {EditableType, GetNextEditableTitles} from '../../models'; +import {camelizeKeys} from 'indico/utils/case'; + import {fileTypePropTypes} from '../../editing/timeline/FileManager/util'; +import {EditableType, GetNextEditableTitles} from '../../models'; import './NextEditable.module.scss'; export default function NextEditable({eventId, editableType, onClose, management}) { const {data: fileTypes, loading: isLoadingFileTypes} = useIndicoAxios({ - url: fileTypesURL({confId: eventId, type: editableType}), + url: fileTypesURL({event_id: eventId, type: editableType}), camelize: true, trigger: [eventId, editableType], }); @@ -67,10 +69,13 @@ function NextEditableDisplay({eventId, editableType, onClose, fileTypes, managem setLoading(true); let response; try { - response = await indicoAxios.post(editableListURL({confId: eventId, type: editableType}), { - extensions: _.pickBy(filters, x => Array.isArray(x)), - has_files: _.pickBy(filters, x => !Array.isArray(x)), - }); + response = await indicoAxios.post( + editableListURL({event_id: eventId, type: editableType}), + { + extensions: _.pickBy(filters, x => Array.isArray(x)), + has_files: _.pickBy(filters, x => !Array.isArray(x)), + } + ); } catch (e) { handleAxiosError(e); setLoading(false); @@ -110,7 +115,7 @@ function NextEditableDisplay({eventId, editableType, onClose, fileTypes, managem try { await indicoAxios.put( assignMyselfURL({ - confId: eventId, + event_id: eventId, contrib_id: selectedEditable.contributionId, type: editableType, }) diff --git a/indico/modules/events/editing/client/js/management/editable_type/NextEditable.module.scss b/indico/modules/events/editing/client/js/management/editable_type/NextEditable.module.scss index d1a637d38ac..28988095a8f 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/NextEditable.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/NextEditable.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/TeamManager.jsx b/indico/modules/events/editing/client/js/management/editable_type/TeamManager.jsx index b638014df2b..9263f7dad0f 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/TeamManager.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/TeamManager.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -8,11 +8,11 @@ import principalsURL from 'indico-url:event_editing.api_editable_type_principals'; import _ from 'lodash'; -import React from 'react'; import PropTypes from 'prop-types'; +import React from 'react'; import {Form as FinalForm} from 'react-final-form'; import {Button, Form, Loader, Message, Modal} from 'semantic-ui-react'; -import {Translate} from 'indico/react/i18n'; + import {FinalPrincipalList} from 'indico/react/components'; import { getChangedValues, @@ -20,23 +20,24 @@ import { FinalSubmitButton, FinalUnloadPrompt, } from 'indico/react/forms'; -import {indicoAxios} from 'indico/utils/axios'; import {useFavoriteUsers, useIndicoAxios} from 'indico/react/hooks'; +import {Translate} from 'indico/react/i18n'; import {useNumericParam} from 'indico/react/util/routing'; +import {indicoAxios} from 'indico/utils/axios'; export default function TeamManager({editableType, onClose}) { - const eventId = useNumericParam('confId'); + const eventId = useNumericParam('event_id'); const favoriteUsersController = useFavoriteUsers(); const {data: principals, loading: isLoadingPrincipals} = useIndicoAxios({ - url: principalsURL({confId: eventId, type: editableType}), + url: principalsURL({event_id: eventId, type: editableType}), trigger: [eventId, editableType], }); const handleSubmit = async (data, form) => { const changedValues = getChangedValues(data, form); try { - await indicoAxios.post(principalsURL({confId: eventId, type: editableType}), changedValues); + await indicoAxios.post(principalsURL({event_id: eventId, type: editableType}), changedValues); } catch (error) { return handleSubmitError(error); } diff --git a/indico/modules/events/editing/client/js/management/editable_type/file_types/ExtensionList.jsx b/indico/modules/events/editing/client/js/management/editable_type/file_types/ExtensionList.jsx index bff98b56be1..8778f5e01ea 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/file_types/ExtensionList.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/file_types/ExtensionList.jsx @@ -1,14 +1,15 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. import _ from 'lodash'; -import React from 'react'; import PropTypes from 'prop-types'; +import React from 'react'; import {Dropdown} from 'semantic-ui-react'; + import {Translate} from 'indico/react/i18n'; export default function ExtensionList({value, disabled, onChange, onFocus, onBlur}) { diff --git a/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.jsx b/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.jsx index dd4543181c5..789779b7e8c 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.jsx @@ -1,25 +1,28 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import fileTypesURL from 'indico-url:event_editing.api_file_types'; import createFileTypeURL from 'indico-url:event_editing.api_add_file_type'; import editFileTypeURL from 'indico-url:event_editing.api_edit_file_type'; +import fileTypesURL from 'indico-url:event_editing.api_file_types'; -import React, {useReducer} from 'react'; import PropTypes from 'prop-types'; +import React, {useReducer} from 'react'; import {Button, Icon, Loader, Message, Segment, Popup, Label} from 'semantic-ui-react'; + import {RequestConfirm, TooltipIfTruncated} from 'indico/react/components'; -import {Param, Translate} from 'indico/react/i18n'; import {getChangedValues, handleSubmitError} from 'indico/react/forms'; import {useIndicoAxios} from 'indico/react/hooks'; +import {Param, Translate} from 'indico/react/i18n'; import {handleAxiosError, indicoAxios} from 'indico/utils/axios'; -import FileTypeModal from './FileTypeModal'; + import {EditableType} from '../../../models'; +import FileTypeModal from './FileTypeModal'; + import './FileTypeManager.module.scss'; const initialState = { @@ -57,14 +60,14 @@ StatusIcon.propTypes = { export default function FileTypeManager({eventId, editableType}) { const [state, dispatch] = useReducer(fileTypesReducer, initialState); const {data, loading: isLoadingFileTypes, reFetch, lastData} = useIndicoAxios({ - url: fileTypesURL({confId: eventId, type: editableType}), + url: fileTypesURL({event_id: eventId, type: editableType}), camelize: true, trigger: eventId, }); const createFileType = async formData => { try { - await indicoAxios.post(createFileTypeURL({confId: eventId, type: editableType}), formData); + await indicoAxios.post(createFileTypeURL({event_id: eventId, type: editableType}), formData); reFetch(); } catch (e) { return handleSubmitError(e); @@ -72,7 +75,7 @@ export default function FileTypeManager({eventId, editableType}) { }; const editFileType = async (fileTypeId, fileTypeData) => { - const url = editFileTypeURL({confId: eventId, file_type_id: fileTypeId, type: editableType}); + const url = editFileTypeURL({event_id: eventId, file_type_id: fileTypeId, type: editableType}); try { await indicoAxios.patch(url, fileTypeData); @@ -83,7 +86,7 @@ export default function FileTypeManager({eventId, editableType}) { }; const deleteFileType = async fileTypeId => { - const url = editFileTypeURL({confId: eventId, file_type_id: fileTypeId, type: editableType}); + const url = editFileTypeURL({event_id: eventId, file_type_id: fileTypeId, type: editableType}); try { await indicoAxios.delete(url); diff --git a/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.module.scss b/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.module.scss index 6e94f4f9720..40faa758f0e 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeManager.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeModal.jsx b/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeModal.jsx index 7d6d280b308..8d5b9bc50a2 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeModal.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/file_types/FileTypeModal.jsx @@ -1,14 +1,15 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import React from 'react'; import PropTypes from 'prop-types'; +import React from 'react'; import {Form as FinalForm} from 'react-final-form'; import {Button, Form, Header, Icon, Modal, Message} from 'semantic-ui-react'; + import { FinalCheckbox, FinalField, @@ -17,6 +18,7 @@ import { unsortedArraysEqual, } from 'indico/react/forms'; import {Param, Translate} from 'indico/react/i18n'; + import ExtensionList from './ExtensionList'; export default function FileTypeModal({onClose, onSubmit, fileType, header}) { diff --git a/indico/modules/events/editing/client/js/management/editable_type/file_types/index.jsx b/indico/modules/events/editing/client/js/management/editable_type/file_types/index.jsx index 641a687db61..1cfeeb22643 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/file_types/index.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/file_types/index.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -7,13 +7,16 @@ import React from 'react'; import {useParams} from 'react-router-dom'; -import {useNumericParam} from 'indico/react/util/routing'; + import {Translate} from 'indico/react/i18n'; +import {useNumericParam} from 'indico/react/util/routing'; + import EditableTypeSubPageNav from '../EditableTypeSubPageNav'; + import FileTypeManager from './FileTypeManager'; export default function FileTypeManagement() { - const eventId = useNumericParam('confId'); + const eventId = useNumericParam('event_id'); const {type} = useParams(); return ( diff --git a/indico/modules/events/editing/client/js/management/editable_type/index.js b/indico/modules/events/editing/client/js/management/editable_type/index.js index 02cb00f7998..533c8e5405e 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/index.js +++ b/indico/modules/events/editing/client/js/management/editable_type/index.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.jsx b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.jsx index aaeeea0802b..f2301842652 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -7,18 +7,19 @@ import editReviewConditionURL from 'indico-url:event_editing.api_edit_review_condition'; -import React, {useState, useContext} from 'react'; import PropTypes from 'prop-types'; +import React, {useState, useContext} from 'react'; import {Icon, Label} from 'semantic-ui-react'; import {RequestConfirm, TooltipIfTruncated} from 'indico/react/components'; -import {Translate} from 'indico/react/i18n'; import {handleSubmitError} from 'indico/react/forms'; +import {Translate} from 'indico/react/i18n'; import {handleAxiosError, indicoAxios} from 'indico/utils/axios'; -import ReviewConditionForm from './ReviewConditionForm'; import {EditableType} from '../../../models'; + import ReviewConditionsContext from './context'; +import ReviewConditionForm from './ReviewConditionForm'; import './ConditionInfo.module.scss'; @@ -26,7 +27,7 @@ export default function ConditionInfo({fileTypes, condId, editableType, onUpdate const [isEditing, setIsEditing] = useState(false); const [isDeleting, setIsDeleting] = useState(false); const {eventId} = useContext(ReviewConditionsContext); - const url = editReviewConditionURL({confId: eventId, condition_id: condId, type: editableType}); + const url = editReviewConditionURL({event_id: eventId, condition_id: condId, type: editableType}); const deleteCondition = async () => { try { diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.module.scss b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.module.scss index d563021fa86..bb5106eb77b 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ConditionInfo.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.jsx b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.jsx index 8fc22d75c06..00627a1853f 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.jsx @@ -1,17 +1,18 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import React, {useContext} from 'react'; import PropTypes from 'prop-types'; +import React, {useContext} from 'react'; import {Form as FinalForm} from 'react-final-form'; import {Button, Form} from 'semantic-ui-react'; import {FinalDropdown, FinalSubmitButton} from 'indico/react/forms'; import {Translate} from 'indico/react/i18n'; + import ReviewConditionsContext from './context'; import './ReviewConditionForm.module.scss'; diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.module.scss b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.module.scss index 5bebe112d63..c1df64bac51 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionForm.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.jsx b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.jsx index 409f756b7c6..07ce77a868b 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -16,8 +16,8 @@ import {Translate} from 'indico/react/i18n'; import {indicoAxios} from 'indico/utils/axios'; import ConditionInfo from './ConditionInfo'; -import ReviewConditionForm from './ReviewConditionForm'; import ReviewConditionsContext from './context'; +import ReviewConditionForm from './ReviewConditionForm'; import './ReviewConditionsManager.module.scss'; @@ -25,7 +25,7 @@ export default function ReviewConditionsManager() { const {eventId, fileTypes, editableType} = useContext(ReviewConditionsContext); const [isAdding, setIsAdding] = useState(false); const {loading, reFetch, data: eventConditionsSetting, lastData} = useIndicoAxios({ - url: reviewConditionsURL({confId: eventId, type: editableType}), + url: reviewConditionsURL({event_id: eventId, type: editableType}), trigger: eventId, }); @@ -43,7 +43,10 @@ export default function ReviewConditionsManager() { ]); const createNewCondition = async formData => { try { - await indicoAxios.post(reviewConditionsURL({confId: eventId, type: editableType}), formData); + await indicoAxios.post( + reviewConditionsURL({event_id: eventId, type: editableType}), + formData + ); setIsAdding(false); reFetch(); } catch (e) { diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.module.scss b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.module.scss index 9c8dd12c0f0..6f58c3267fc 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.module.scss +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/ReviewConditionsManager.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/context.js b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/context.js index 6c9382ab9a0..3ea9ec5184b 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/context.js +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/context.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/index.jsx b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/index.jsx index fa59e93680f..9115910e4a3 100644 --- a/indico/modules/events/editing/client/js/management/editable_type/review_conditions/index.jsx +++ b/indico/modules/events/editing/client/js/management/editable_type/review_conditions/index.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -9,19 +9,22 @@ import fileTypesURL from 'indico-url:event_editing.api_file_types'; import React from 'react'; import {useParams} from 'react-router-dom'; -import {useNumericParam} from 'indico/react/util/routing'; -import {Translate} from 'indico/react/i18n'; + import {useIndicoAxios} from 'indico/react/hooks'; +import {Translate} from 'indico/react/i18n'; +import {useNumericParam} from 'indico/react/util/routing'; + import EditableTypeSubPageNav from '../EditableTypeSubPageNav'; -import ReviewConditionsManager from './ReviewConditionsManager'; + import ReviewConditionsContext from './context'; +import ReviewConditionsManager from './ReviewConditionsManager'; export default function ReviewConditionManagement() { - const eventId = useNumericParam('confId'); + const eventId = useNumericParam('event_id'); const {type} = useParams(); const {data: fileTypes} = useIndicoAxios({ - url: fileTypesURL({confId: eventId, type}), + url: fileTypesURL({event_id: eventId, type}), camelize: true, trigger: [eventId, type], }); diff --git a/indico/modules/events/editing/client/js/management/index.jsx b/indico/modules/events/editing/client/js/management/index.jsx index 0080c1d23b4..cec0b03b182 100644 --- a/indico/modules/events/editing/client/js/management/index.jsx +++ b/indico/modules/events/editing/client/js/management/index.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/tags/TagManager.jsx b/indico/modules/events/editing/client/js/management/tags/TagManager.jsx index 6d136ce02f7..fc1871cd02a 100644 --- a/indico/modules/events/editing/client/js/management/tags/TagManager.jsx +++ b/indico/modules/events/editing/client/js/management/tags/TagManager.jsx @@ -1,23 +1,24 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import tagsURL from 'indico-url:event_editing.api_tags'; import createTagURL from 'indico-url:event_editing.api_create_tag'; import editTagURL from 'indico-url:event_editing.api_edit_tag'; +import tagsURL from 'indico-url:event_editing.api_tags'; -import React, {useReducer} from 'react'; import PropTypes from 'prop-types'; +import React, {useReducer} from 'react'; import {Button, Icon, Label, Loader, Message, Segment, Popup} from 'semantic-ui-react'; -import {Param, Translate} from 'indico/react/i18n'; import {RequestConfirm} from 'indico/react/components'; import {getChangedValues, handleSubmitError} from 'indico/react/forms'; import {useIndicoAxios} from 'indico/react/hooks'; +import {Param, Translate} from 'indico/react/i18n'; import {handleAxiosError, indicoAxios} from 'indico/utils/axios'; + import TagModal from './TagModal'; import './TagManager.module.scss'; @@ -45,14 +46,14 @@ function tagsReducer(state, action) { export default function TagManager({eventId}) { const [state, dispatch] = useReducer(tagsReducer, initialState); const {data, loading: isLoadingTags, reFetch, lastData} = useIndicoAxios({ - url: tagsURL({confId: eventId}), + url: tagsURL({event_id: eventId}), camelize: true, trigger: eventId, }); const createTag = async formData => { try { - await indicoAxios.post(createTagURL({confId: eventId}), formData); + await indicoAxios.post(createTagURL({event_id: eventId}), formData); reFetch(); } catch (e) { return handleSubmitError(e); @@ -60,7 +61,7 @@ export default function TagManager({eventId}) { }; const editTag = async (tagId, tagData) => { - const url = editTagURL({confId: eventId, tag_id: tagId}); + const url = editTagURL({event_id: eventId, tag_id: tagId}); try { await indicoAxios.patch(url, tagData); @@ -71,7 +72,7 @@ export default function TagManager({eventId}) { }; const deleteTag = async tagId => { - const url = editTagURL({confId: eventId, tag_id: tagId}); + const url = editTagURL({event_id: eventId, tag_id: tagId}); try { await indicoAxios.delete(url); diff --git a/indico/modules/events/editing/client/js/management/tags/TagManager.module.scss b/indico/modules/events/editing/client/js/management/tags/TagManager.module.scss index 0f3fa04b02d..a06647cfac0 100644 --- a/indico/modules/events/editing/client/js/management/tags/TagManager.module.scss +++ b/indico/modules/events/editing/client/js/management/tags/TagManager.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/client/js/management/tags/TagModal.jsx b/indico/modules/events/editing/client/js/management/tags/TagModal.jsx index 968760c206b..52f0f9648fb 100644 --- a/indico/modules/events/editing/client/js/management/tags/TagModal.jsx +++ b/indico/modules/events/editing/client/js/management/tags/TagModal.jsx @@ -1,12 +1,12 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import React from 'react'; import PropTypes from 'prop-types'; +import React from 'react'; import {Form as FinalForm} from 'react-final-form'; import {Button, Form, Label, Modal} from 'semantic-ui-react'; diff --git a/indico/modules/events/editing/client/js/management/tags/index.js b/indico/modules/events/editing/client/js/management/tags/index.js index 4be0c37d951..fc41ad842d1 100644 --- a/indico/modules/events/editing/client/js/management/tags/index.js +++ b/indico/modules/events/editing/client/js/management/tags/index.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -8,16 +8,18 @@ import dashboardURL from 'indico-url:event_editing.dashboard'; import React from 'react'; + import {ManagementPageBackButton, ManagementPageSubTitle} from 'indico/react/components'; import {Translate} from 'indico/react/i18n'; import {useNumericParam} from 'indico/react/util/routing'; + import TagManager from './TagManager'; export default function EditingTagManagement() { - const eventId = useNumericParam('confId'); + const eventId = useNumericParam('event_id'); return ( <> - + diff --git a/indico/modules/events/editing/client/js/models.js b/indico/modules/events/editing/client/js/models.js index 8cb59e1771a..1fa3dfa354c 100644 --- a/indico/modules/events/editing/client/js/models.js +++ b/indico/modules/events/editing/client/js/models.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/clone.py b/indico/modules/events/editing/clone.py index c476a6f0435..1a3cf0628da 100644 --- a/indico/modules/events/editing/clone.py +++ b/indico/modules/events/editing/clone.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy.util.models import get_simple_column_attrs from indico.core.db.sqlalchemy.util.session import no_autoflush diff --git a/indico/modules/events/editing/controllers/backend/common.py b/indico/modules/events/editing/controllers/backend/common.py index 219a262a857..d492ab68ae0 100644 --- a/indico/modules/events/editing/controllers/backend/common.py +++ b/indico/modules/events/editing/controllers/backend/common.py @@ -1,13 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - -from flask import jsonify, request +from flask import jsonify, request, session from indico.core import signals from indico.modules.events.editing.controllers.base import RHEditableTypeEditorBase, RHEditingBase @@ -46,7 +44,16 @@ class RHMenuEntries(RHEditingBase): def _process(self): menu_entries = named_objects_from_signal(signals.menu.items.send('event-editing-sidemenu', event=self.event)) - return EditingMenuItemSchema(many=True).jsonify(menu_entries.values()) + is_editing_manager = self.event.can_manage(session.user, permission='editing_manager') + show_editable_list = { + et.name: (is_editing_manager or self.event.can_manage(session.user, et.editor_permission)) + for et in EditableType + } + return jsonify( + items=EditingMenuItemSchema(many=True).dump(list(menu_entries.values())), + show_management_link=is_editing_manager, + show_editable_list=show_editable_list + ) class RHEditableCheckSelfAssign(RHEditableTypeEditorBase): diff --git a/indico/modules/events/editing/controllers/backend/editable_list.py b/indico/modules/events/editing/controllers/backend/editable_list.py index 5100cf65ea7..a25ba92c256 100644 --- a/indico/modules/events/editing/controllers/backend/editable_list.py +++ b/indico/modules/events/editing/controllers/backend/editable_list.py @@ -1,29 +1,28 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import uuid from flask import jsonify, request, session from marshmallow import fields -from sqlalchemy.orm import joinedload +from sqlalchemy.orm import joinedload, selectinload from sqlalchemy.sql import and_, func, over from werkzeug.exceptions import Forbidden +from indico.core.cache import make_scoped_cache from indico.core.db import db -from indico.legacy.common.cache import GenericCache from indico.modules.events.contributions.models.contributions import Contribution from indico.modules.events.editing.controllers.base import (RHEditablesBase, RHEditableTypeEditorBase, RHEditableTypeManagementBase) from indico.modules.events.editing.models.editable import Editable from indico.modules.events.editing.models.revision_files import EditingRevisionFile from indico.modules.events.editing.models.revisions import EditingRevision -from indico.modules.events.editing.operations import assign_editor, generate_editables_zip, unassign_editor +from indico.modules.events.editing.operations import (assign_editor, generate_editables_json, generate_editables_zip, + unassign_editor) from indico.modules.events.editing.schemas import EditableBasicSchema, EditingEditableListSchema, FilteredEditableSchema from indico.modules.files.models.files import File from indico.util.i18n import _ @@ -32,11 +31,11 @@ from indico.web.flask.util import url_for -archive_cache = GenericCache('editables-archive') +archive_cache = make_scoped_cache('editables-archive') class RHEditableList(RHEditableTypeEditorBase): - """Return the list of editables of the event for a given type""" + """Return the list of editables of the event for a given type.""" def _process_args(self): RHEditableTypeEditorBase._process_args(self) self.contributions = (Contribution.query @@ -52,18 +51,33 @@ def _process(self): class RHPrepareEditablesArchive(RHEditablesBase): def _process(self): - key = unicode(uuid.uuid4()) + key = str(uuid.uuid4()) data = [editable.id for editable in self.editables] - archive_cache.set(key, data, time=1800) - download_url = url_for('.download_archive', self.event, type=self.editable_type.name, uuid=key) + archive_cache.set(key, data, timeout=1800) + archive_type = request.view_args['archive_type'] + download_url = url_for('.download_archive', self.event, type=self.editable_type.name, + archive_type=archive_type, uuid=key) return jsonify(download_url=download_url) class RHDownloadArchive(RHEditableTypeManagementBase): def _process(self): - editable_ids = archive_cache.get(unicode(request.view_args['uuid']), []) - editables = Editable.query.filter(Editable.id.in_(editable_ids)).all() - return generate_editables_zip(editables) + editable_ids = archive_cache.get(str(request.view_args['uuid']), []) + revisions_strategy = selectinload('revisions') + revisions_strategy.subqueryload('comments').joinedload('user') + revisions_strategy.subqueryload('files').joinedload('file_type') + revisions_strategy.subqueryload('tags') + revisions_strategy.joinedload('submitter') + revisions_strategy.joinedload('editor') + editables = (Editable.query + .filter(Editable.id.in_(editable_ids)) + .options(joinedload('editor'), joinedload('contribution'), revisions_strategy) + .all()) + fn = { + 'archive': generate_editables_zip, + 'json': generate_editables_json, + }[request.view_args['archive_type']] + return fn(self.event, self.editable_type, editables) class RHAssignEditor(RHEditablesBase): diff --git a/indico/modules/events/editing/controllers/backend/management.py b/indico/modules/events/editing/controllers/backend/management.py index 0a8ea1bdf5d..805e65b8768 100644 --- a/indico/modules/events/editing/controllers/backend/management.py +++ b/indico/modules/events/editing/controllers/backend/management.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import jsonify, request, session from werkzeug.exceptions import Forbidden @@ -180,6 +178,19 @@ def _process_DELETE(self): return '', 204 +class RHEditableSetAnonymousTeam(RHEditableTypeManagementBase): + def _process_GET(self): + return jsonify(editable_type_settings[self.editable_type].get(self.event, 'anonymous_team')) + + def _process_PUT(self): + editable_type_settings[self.editable_type].set(self.event, 'anonymous_team', True) + return '', 204 + + def _process_DELETE(self): + editable_type_settings[self.editable_type].set(self.event, 'anonymous_team', False) + return '', 204 + + class RHEditableTypePrincipals(RHEditableTypeManagementBase): def _process_GET(self): permission_name = self.editable_type.editor_permission @@ -210,13 +221,13 @@ def _process_GET(self): def _process_PUT(self): self.event.log(EventLogRealm.management, EventLogKind.positive, 'Editing', - 'Opened {} submission'.format(orig_string(self.editable_type.title)), session.user) + f'Opened {orig_string(self.editable_type.title)} submission', session.user) editable_type_settings[self.editable_type].set(self.event, 'submission_enabled', True) return '', 204 def _process_DELETE(self): self.event.log(EventLogRealm.management, EventLogKind.negative, 'Editing', - 'Closed {} submission'.format(orig_string(self.editable_type.title)), session.user) + f'Closed {orig_string(self.editable_type.title)} submission', session.user) editable_type_settings[self.editable_type].set(self.event, 'submission_enabled', False) return '', 204 @@ -227,13 +238,13 @@ def _process_GET(self): def _process_PUT(self): self.event.log(EventLogRealm.management, EventLogKind.positive, 'Editing', - 'Opened {} editing'.format(orig_string(self.editable_type.title)), session.user) + f'Opened {orig_string(self.editable_type.title)} editing', session.user) editable_type_settings[self.editable_type].set(self.event, 'editing_enabled', True) return '', 204 def _process_DELETE(self): self.event.log(EventLogRealm.management, EventLogKind.negative, 'Editing', - 'Closed {} editing'.format(orig_string(self.editable_type.title)), session.user) + f'Closed {orig_string(self.editable_type.title)} editing', session.user) editable_type_settings[self.editable_type].set(self.event, 'editing_enabled', False) return '', 204 diff --git a/indico/modules/events/editing/controllers/backend/service.py b/indico/modules/events/editing/controllers/backend/service.py index 1df1c2beb02..6bf302bf45b 100644 --- a/indico/modules/events/editing/controllers/backend/service.py +++ b/indico/modules/events/editing/controllers/backend/service.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from uuid import uuid4 from flask import jsonify @@ -30,7 +28,7 @@ class RHCheckServiceURL(RHEditingManagementBase): @use_kwargs({ 'url': fields.URL(schemes={'http', 'https'}, required=True), - }) + }, location='query') def _process(self, url): url = url.rstrip('/') return jsonify(check_service_url(url)) @@ -43,7 +41,7 @@ class RHConnectService(RHEditingManagementBase): 'url': fields.URL(schemes={'http', 'https'}, required=True), }) def _process(self, url): - if not config.DEBUG: + if not config.EXPERIMENTAL_EDITING_SERVICE: raise ServiceUnavailable('This functionality is not available yet') if editing_settings.get(self.event, 'service_url'): raise BadRequest('Service URL already set') @@ -55,7 +53,7 @@ def _process(self, url): editing_settings.set(self.event, 'service_event_identifier', make_event_identifier(self.event)) editing_settings.set_multi(self.event, { 'service_url': url, - 'service_token': unicode(uuid4()), + 'service_token': str(uuid4()), }) # we need to commit the token so the service can already use it when processing # the enabled event in case it wants to set up tags etc @@ -102,7 +100,7 @@ def _process(self, force): class RHServiceStatus(RHEditingManagementBase): - """Get the status of the currently connected service""" + """Get the status of the currently connected service.""" def _process(self): if not editing_settings.get(self.event, 'service_url'): diff --git a/indico/modules/events/editing/controllers/backend/timeline.py b/indico/modules/events/editing/controllers/backend/timeline.py index a534a3f8e9d..49fb94b29ac 100644 --- a/indico/modules/events/editing/controllers/backend/timeline.py +++ b/indico/modules/events/editing/controllers/backend/timeline.py @@ -1,23 +1,20 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os from io import BytesIO from zipfile import ZipFile -from flask import request, session -from marshmallow import fields +from flask import jsonify, request, session +from marshmallow import EXCLUDE, fields from marshmallow_enum import EnumField from sqlalchemy.orm import joinedload from werkzeug.exceptions import Forbidden, NotFound, ServiceUnavailable -from indico.core.db import db from indico.core.errors import UserValueError from indico.modules.events.editing.controllers.base import RHContributionEditableBase, TokenAccessMixin from indico.modules.events.editing.fields import EditingFilesField, EditingTagsField @@ -26,12 +23,15 @@ from indico.modules.events.editing.models.revisions import EditingRevision, InitialRevisionState from indico.modules.events.editing.operations import (assign_editor, confirm_editable_changes, create_new_editable, create_revision_comment, create_submitter_revision, - delete_editable, delete_revision_comment, replace_revision, + delete_revision_comment, ensure_latest_revision, + publish_editable_revision, replace_revision, review_editable_revision, unassign_editor, undo_review, update_revision_comment) from indico.modules.events.editing.schemas import (EditableSchema, EditingConfirmationAction, EditingReviewAction, ReviewEditableArgs) -from indico.modules.events.editing.service import ServiceRequestFailed, service_handle_new_editable +from indico.modules.events.editing.service import (ServiceRequestFailed, service_get_custom_actions, + service_handle_custom_action, service_handle_new_editable, + service_handle_review_editable) from indico.modules.events.editing.settings import editing_settings from indico.modules.files.controllers import UploadFileMixin from indico.modules.users import User @@ -81,6 +81,7 @@ def _process_args(self): .with_parent(self.editable, 'revisions') .filter_by(id=request.view_args['revision_id']) .first_or_404()) + if self.revision is None: raise NotFound @@ -110,8 +111,44 @@ def _check_access(self): if not self.editable.can_see_timeline(session.user): raise Forbidden + def _get_custom_actions(self): + if not editing_settings.get(self.event, 'service_url'): + return [] + + try: + return service_get_custom_actions(self.editable, self.editable.revisions[-1], session.user) + except ServiceRequestFailed: + # unlikely to fail, but if it does we don't break the whole timeline + return [] + def _process(self): - return EditableSchema(context={'user': session.user}).jsonify(self.editable) + custom_actions = self._get_custom_actions() + custom_actions_ctx = {self.editable.revisions[-1]: custom_actions} + schema = EditableSchema(context={ + 'user': session.user, + 'custom_actions': custom_actions_ctx, + 'can_see_editor_names': self.editable.can_see_editor_names, + }) + return schema.jsonify(self.editable) + + +class RHTriggerExtraRevisionAction(RHContributionEditableRevisionBase): + """Trigger an extra action provided by the editing service.""" + + def _check_revision_access(self): + if not editing_settings.get(self.event, 'service_url'): + return False + # It's up to the editing service to decide who can do what, so we + # just require the user to have editable access + return self.editable.can_see_timeline(session.user) + + @use_kwargs({ + 'action': fields.String(required=True) + }) + def _process(self, action): + ensure_latest_revision(self.revision) + resp = service_handle_custom_action(self.editable, self.revision, session.user, action) + return jsonify(redirect=resp.get('redirect')) class RHCreateEditable(RHContributionEditableBase): @@ -138,14 +175,12 @@ def _process(self): initial_state = InitialRevisionState.new if service_url else InitialRevisionState.ready_for_review editable = create_new_editable(self.contrib, self.editable_type, session.user, args['files'], initial_state) - db.session.commit() if service_url: try: - service_handle_new_editable(editable) + service_handle_new_editable(editable, session.user) except ServiceRequestFailed: - delete_editable(editable) - db.session.commit() raise ServiceUnavailable(_('Submission failed, please try again later.')) + return '', 201 @@ -161,8 +196,22 @@ def _process(self, action, comment): if action in (EditingReviewAction.update, EditingReviewAction.update_accept): argmap['files'] = EditingFilesField(self.event, self.contrib, self.editable_type, allow_claimed_files=True, required=True) - args = parser.parse(argmap) - review_editable_revision(self.revision, session.user, action, comment, args['tags'], args.get('files')) + args = parser.parse(argmap, unknown=EXCLUDE) + service_url = editing_settings.get(self.event, 'service_url') + + new_revision = review_editable_revision(self.revision, session.user, action, comment, args['tags'], + args.get('files')) + + publish = True + if service_url: + try: + resp = service_handle_review_editable(self.editable, session.user, action, self.revision, new_revision) + publish = resp.get('publish', True) + except ServiceRequestFailed: + raise ServiceUnavailable(_('Failed processing review, please try again later.')) + + if publish and action in (EditingReviewAction.accept, EditingReviewAction.update_accept): + publish_editable_revision(new_revision or self.revision) return '', 204 @@ -178,6 +227,18 @@ def _check_revision_access(self): }) def _process(self, action, comment): confirm_editable_changes(self.revision, session.user, action, comment) + + service_url = editing_settings.get(self.event, 'service_url') + publish = True + if service_url: + try: + resp = service_handle_review_editable(self.editable, session.user, action, self.revision) + publish = resp.get('publish', True) + except ServiceRequestFailed: + raise ServiceUnavailable(_('Failed processing review, please try again later.')) + + if publish and action == EditingConfirmationAction.accept: + publish_editable_revision(self.revision) return '', 204 @@ -199,7 +260,7 @@ def _process(self, comment, state): 'tags': EditingTagsField(self.event, allow_system_tags=self.is_service_call, missing=set()), 'files': EditingFilesField(self.event, self.contrib, self.editable_type, allow_claimed_files=True, required=True) - }) + }, unknown=EXCLUDE) user = User.get_system_user() if self.is_service_call else session.user replace_revision(self.revision, user, comment, args['files'], args['tags'], state) @@ -218,13 +279,23 @@ def _process(self): required=True) }) - create_submitter_revision(self.revision, session.user, args['files']) + service_url = editing_settings.get(self.event, 'service_url') + new_revision = create_submitter_revision(self.revision, session.user, args['files']) + + if service_url: + try: + service_handle_review_editable(self.editable, session.user, EditingReviewAction.update, + self.revision, new_revision) + except ServiceRequestFailed: + raise ServiceUnavailable(_('Failed processing review, please try again later.')) return '', 204 class RHUndoReview(RHContributionEditableRevisionBase): """Undo the last review/confirmation on an Editable.""" + SERVICE_ALLOWED = True + def _check_revision_access(self): return self.editable.can_perform_editor_actions(session.user) @@ -234,7 +305,9 @@ def _process(self): class RHCreateRevisionComment(RHContributionEditableRevisionBase): - """Create new revision comment""" + """Create new revision comment.""" + + SERVICE_ALLOWED = True def _check_revision_access(self): return self.editable.can_comment(session.user) @@ -244,9 +317,13 @@ def _check_revision_access(self): 'internal': fields.Bool(missing=False) }) def _process(self, text, internal): - if internal and not self.editable.can_use_internal_comments(session.user): + user = session.user + if self.is_service_call: + user = User.get_system_user() + elif internal and not self.editable.can_use_internal_comments(session.user): internal = False - create_revision_comment(self.revision, session.user, text, internal) + + create_revision_comment(self.revision, user, text, internal) return '', 201 @@ -289,7 +366,7 @@ def _process_DELETE(self): class RHExportRevisionFiles(RHContributionEditableRevisionBase): - """Export revision files as a ZIP archive""" + """Export revision files as a ZIP archive.""" def _check_revision_access(self): return self.editable.can_see_timeline(session.user) @@ -299,19 +376,19 @@ def _process(self): with ZipFile(buf, 'w', allowZip64=True) as zip_handler: for revision_file in self.revision.files: file = revision_file.file - filename = secure_filename(file.filename, 'file-{}'.format(file.id)) + filename = secure_filename(file.filename, f'file-{file.id}') file_type = revision_file.file_type - folder_name = secure_filename(file_type.name, 'file-type-{}'.format(file_type.id)) + folder_name = secure_filename(file_type.name, f'file-type-{file_type.id}') with file.storage.get_local_path(file.storage_file_id) as filepath: zip_handler.write(filepath, os.path.join(folder_name, filename)) buf.seek(0) - return send_file('revision-{}.zip'.format(self.revision.id), buf, 'application/zip', inline=False) + return send_file(f'revision-{self.revision.id}.zip', buf, 'application/zip', inline=False) class RHDownloadRevisionFile(RHContributionEditableRevisionBase): - """Download a revision file""" + """Download a revision file.""" SERVICE_ALLOWED = True diff --git a/indico/modules/events/editing/controllers/base.py b/indico/modules/events/editing/controllers/base.py index 0168e5f30db..01c8b2904fd 100644 --- a/indico/modules/events/editing/controllers/base.py +++ b/indico/modules/events/editing/controllers/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request, session from werkzeug.exceptions import NotFound, Unauthorized @@ -22,14 +20,14 @@ from indico.web.rh import RequireUserMixin -class TokenAccessMixin(object): +class TokenAccessMixin: SERVICE_ALLOWED = False is_service_call = False def _token_can_access(self): # we need to "fish" the event here because at this point _check_params # hasn't run yet - event = Event.get_or_404(int(request.view_args['confId']), is_deleted=False) + event = Event.get_or_404(request.view_args['event_id'], is_deleted=False) if not self.SERVICE_ALLOWED or not request.bearer_token: return False @@ -43,7 +41,7 @@ def _token_can_access(self): def _check_csrf(self): # check CSRF if there is no bearer token or there's a session cookie if session.user or not request.bearer_token: - super(TokenAccessMixin, self)._check_csrf() + super()._check_csrf() class RHEditingBase(TokenAccessMixin, RequireUserMixin, RHDisplayEventBase): @@ -65,7 +63,7 @@ class RHEditingManagementBase(TokenAccessMixin, RHManageEventBase): def _check_access(self): if not TokenAccessMixin._token_can_access(self): - super(RHEditingManagementBase, self)._check_access() + super()._check_access() class RHEditableTypeManagementBase(RHEditingManagementBase): @@ -106,6 +104,11 @@ def _check_access(self): RequireUserMixin._check_access(self) RHContributionDisplayBase._check_access(self) + def _can_view_unpublished(self): + if super()._can_view_unpublished(): + return True + return self.editable is not None and self.editable.can_see_timeline(session.user) + def _process_args(self): RHContributionDisplayBase._process_args(self) self.editable_type = EditableType[request.view_args['type']] diff --git a/indico/modules/events/editing/controllers/frontend.py b/indico/modules/events/editing/controllers/frontend.py index 48393856471..5900fb5aa7f 100644 --- a/indico/modules/events/editing/controllers/frontend.py +++ b/indico/modules/events/editing/controllers/frontend.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from werkzeug.exceptions import Forbidden, NotFound @@ -20,7 +18,7 @@ class RHEditingDashboard(RHEditingManagementBase): def _process(self): template = 'editing.html' if self.event.has_feature('editing') else 'disabled.html' - return WPEditing.render_template('management/{}'.format(template), self.event) + return WPEditing.render_template(f'management/{template}', self.event) class RHEditableTimeline(RHContributionEditableBase): diff --git a/indico/modules/events/editing/fields.py b/indico/modules/events/editing/fields.py index c9c343cb02e..ecf56851cd2 100644 --- a/indico/modules/events/editing/fields.py +++ b/indico/modules/events/editing/fields.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import fnmatch import os @@ -29,22 +27,22 @@ def __init__(self, event, contrib, editable_type, allow_claimed_files=False, **k keys_field = ModelField(EditingFileType, get_query=lambda m: self.editing_file_types_query) values_field = FilesField(required=True, allow_claimed=allow_claimed_files) validators = kwargs.pop('validate', []) + [self.validate_files] - super(EditingFilesField, self).__init__(keys=keys_field, values=values_field, validate=validators, **kwargs) + super().__init__(keys=keys_field, values=values_field, validate=validators, **kwargs) def validate_files(self, value): required_types = {ft for ft in self.editing_file_types_query if ft.required} # ensure all required file types have files - required_missing = required_types - {ft for ft, files in value.viewitems() if files} + required_missing = required_types - {ft for ft, files in value.items() if files} if required_missing: raise ValidationError('Required file types missing: {}' .format(', '.join(ft.name for ft in required_missing))) seen = set() - for file_type, files in value.viewitems(): + for file_type, files in value.items(): # ensure single-file types don't have too many files if not file_type.allow_multiple_files and len(files) > 1: - raise ValidationError('File type "{}" allows only one file'.format(file_type.name)) + raise ValidationError(f'File type "{file_type.name}" allows only one file') # ensure all files have allowed extensions valid_extensions = {ext.lower() for ext in file_type.extensions} @@ -67,7 +65,7 @@ def validate_files(self, value): duplicates = set(files) & seen if duplicates: raise ValidationError('Files found in multiple types: {}' - .format(', '.join(unicode(f.uuid) for f in duplicates))) + .format(', '.join(str(f.uuid) for f in duplicates))) seen |= set(files) @@ -80,7 +78,7 @@ def _get_query(m): query = query.filter_by(system=False) return query - super(EditingTagsField, self).__init__(model=EditingTag, get_query=_get_query, collection_class=set, **kwargs) + super().__init__(model=EditingTag, get_query=_get_query, collection_class=set, **kwargs) class EditableList(ModelList): @@ -89,4 +87,4 @@ def _get_query(m): return (m.query .join(Contribution) .filter(~Contribution.is_deleted, Contribution.event_id == event.id, m.type == editable_type)) - super(EditableList, self).__init__(model=Editable, get_query=_get_query, collection_class=set, **kwargs) + super().__init__(model=Editable, get_query=_get_query, collection_class=set, **kwargs) diff --git a/indico/modules/events/editing/models/comments.py b/indico/modules/events/editing/models/comments.py index fcc130b2f4c..83da9eda6a3 100644 --- a/indico/modules/events/editing/models/comments.py +++ b/indico/modules/events/editing/models/comments.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy import UTCDateTime from indico.core.db.sqlalchemy.descriptions import RenderMode, RenderModeMixin from indico.util.date_time import now_utc from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii, text_to_repr +from indico.util.string import format_repr, text_to_repr class EditingRevisionComment(RenderModeMixin, db.Model): @@ -93,7 +91,6 @@ class EditingRevisionComment(RenderModeMixin, db.Model): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'revision_id', 'user_id', internal=False, _text=text_to_repr(self.text)) diff --git a/indico/modules/events/editing/models/editable.py b/indico/modules/events/editing/models/editable.py index 991cfe7f599..38d7abbf21a 100644 --- a/indico/modules/events/editing/models/editable.py +++ b/indico/modules/events/editing/models/editable.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy import orm from sqlalchemy.event import listens_for from sqlalchemy.orm import column_property @@ -14,10 +12,10 @@ from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum +from indico.util.enum import RichIntEnum from indico.util.i18n import _ from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_repr from indico.web.flask.util import url_for @@ -100,7 +98,6 @@ class Editable(db.Model): # relationship backrefs: # - revisions (EditingRevision.editable) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'contribution_id', 'type') @@ -180,6 +177,23 @@ def can_use_internal_comments(self, user): """Whether the user can create/see internal comments.""" return self._has_general_editor_permissions(user) + def can_see_editor_names(self, user, actor=None): + """Whether the user can see the names of editing team members. + + This is always true if team anonymity is not enabled; otherwise only + users who are member of the editing team will see names. + + If an `actor` is set, the check applies to whether the name of this + particular user can be seen. + """ + from indico.modules.events.editing.settings import editable_type_settings + + return ( + not editable_type_settings[self.type].get(self.event, 'anonymous_team') or + (actor and not self.can_see_editor_names(actor)) or + self._has_general_editor_permissions(user) + ) + def can_comment(self, user): """Whether the user can comment on the editable.""" # We allow any user associated with the contribution to comment, even if they are @@ -239,7 +253,7 @@ def log(self, *args, **kwargs): @listens_for(orm.mapper, 'after_configured', once=True) def _mappers_configured(): - from .revisions import EditingRevision, InitialRevisionState, FinalRevisionState + from .revisions import EditingRevision, FinalRevisionState, InitialRevisionState # Editable.state -- the state of the editable itself cases = db.cast(db.case({ @@ -260,11 +274,13 @@ def _mappers_configured(): .where(EditingRevision.editable_id == Editable.id) .order_by(EditingRevision.created_dt.desc()) .limit(1) - .correlate_except(EditingRevision)) + .correlate_except(EditingRevision) + .scalar_subquery()) Editable.state = column_property(query) # Editable.revision_count -- the number of revisions the editable has query = (select([db.func.count(EditingRevision.id)]) .where(EditingRevision.editable_id == Editable.id) - .correlate_except(EditingRevision)) + .correlate_except(EditingRevision) + .scalar_subquery()) Editable.revision_count = column_property(query) diff --git a/indico/modules/events/editing/models/file_types.py b/indico/modules/events/editing/models/file_types.py index 56385455d40..e5d2814c60d 100644 --- a/indico/modules/events/editing/models/file_types.py +++ b/indico/modules/events/editing/models/file_types.py @@ -1,19 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import ARRAY from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum from indico.modules.events.editing.models.editable import EditableType -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class EditingFileType(db.Model): @@ -88,7 +86,6 @@ def __table_args__(cls): # - files (EditingRevisionFile.file_type) # - review_conditions (EditingReviewCondition.file_types) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'event_id', 'extensions', allow_multiple_files=False, required=False, publishable=False, filename_template=None, _text=self.name) diff --git a/indico/modules/events/editing/models/review_conditions.py b/indico/modules/events/editing/models/review_conditions.py index 8201095087f..d115241681b 100644 --- a/indico/modules/events/editing/models/review_conditions.py +++ b/indico/modules/events/editing/models/review_conditions.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum from indico.modules.events.editing.models.editable import EditableType -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class EditingReviewCondition(db.Model): @@ -56,7 +54,6 @@ class EditingReviewCondition(db.Model): # relationship backrefs: # - file_types (EditingFileType.review_conditions) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'event_id') diff --git a/indico/modules/events/editing/models/revision_files.py b/indico/modules/events/editing/models/revision_files.py index cce1e7bebfc..283c344439f 100644 --- a/indico/modules/events/editing/models/revision_files.py +++ b/indico/modules/events/editing/models/revision_files.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.util.fs import secure_filename from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr from indico.web.flask.util import url_for @@ -61,14 +59,13 @@ class EditingRevisionFile(db.Model): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'revision_id', 'file_id') @locator_property def locator(self): return dict(self.revision.locator, file_id=self.file_id, - filename=secure_filename(self.file.filename, 'file-{}'.format(self.file_id))) + filename=secure_filename(self.file.filename, f'file-{self.file_id}')) @property def download_url(self): diff --git a/indico/modules/events/editing/models/revisions.py b/indico/modules/events/editing/models/revisions.py index a23aa7c9d18..a05583cbda2 100644 --- a/indico/modules/events/editing/models/revisions.py +++ b/indico/modules/events/editing/models/revisions.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import re from collections import defaultdict @@ -14,10 +12,10 @@ from indico.core.db.sqlalchemy import PyIntEnum, UTCDateTime from indico.core.db.sqlalchemy.descriptions import RenderMode, RenderModeMixin from indico.util.date_time import now_utc +from indico.util.enum import RichIntEnum from indico.util.i18n import _ from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_repr class InitialRevisionState(RichIntEnum): @@ -159,7 +157,6 @@ class EditingRevision(RenderModeMixin, db.Model): # - comments (EditingRevisionComment.revision) # - files (EditingRevisionFile.revision) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'editable_id', 'initial_state', final_state=FinalRevisionState.none) diff --git a/indico/modules/events/editing/models/tags.py b/indico/modules/events/editing/models/tags.py index adac196e15d..165c672b913 100644 --- a/indico/modules/events/editing/models/tags.py +++ b/indico/modules/events/editing/models/tags.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.ext.declarative import declared_attr from indico.core.db import db -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class EditingTag(db.Model): @@ -66,9 +64,8 @@ def __table_args__(cls): @property def verbose_title(self): """Properly formatted title, including tag code.""" - return '{}: {}'.format(self.code, self.title) + return f'{self.code}: {self.title}' - @return_ascii def __repr__(self): return format_repr(self, 'id', 'event_id', system=False, _text=self.title) diff --git a/indico/modules/events/editing/notifications.py b/indico/modules/events/editing/notifications.py index 5e49b2a6b90..8b641d91119 100644 --- a/indico/modules/events/editing/notifications.py +++ b/indico/modules/events/editing/notifications.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -39,8 +39,9 @@ def notify_comment(comment): recipients.discard(None) # in case there's no editor assigned recipients.discard(author) # never bother people about their own comments for recipient in recipients: + author_name = author.first_name if revision.editable.can_see_editor_names(recipient, author) else None tpl = get_template_module('events/editing/emails/comment_notification.txt', - author_name=author.first_name, + author_name=author_name, timeline_url=revision.editable.external_timeline_url, recipient_name=recipient.first_name) send_email(make_email(recipient.email, template=tpl)) @@ -49,8 +50,9 @@ def notify_comment(comment): def notify_editor_judgment(revision, editor): """Notify the submitter about a judgment made by an editor.""" submitter = revision.submitter + editor_name = editor.first_name if revision.editable.can_see_editor_names(submitter) else None tpl = get_template_module('events/editing/emails/editor_judgment_notification.txt', - editor_name=editor.first_name, + editor_name=editor_name, timeline_url=revision.editable.external_timeline_url, recipient_name=submitter.first_name) send_email(make_email(submitter.email, template=tpl)) diff --git a/indico/modules/events/editing/operations.py b/indico/modules/events/editing/operations.py index 0a5d60354b1..6b9beed7d21 100644 --- a/indico/modules/events/editing/operations.py +++ b/indico/modules/events/editing/operations.py @@ -1,17 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os from io import BytesIO from zipfile import ZipFile -from flask import session +from flask import jsonify, session from werkzeug.exceptions import BadRequest from indico.core.db import db @@ -26,7 +24,8 @@ from indico.modules.events.editing.models.tags import EditingTag from indico.modules.events.editing.notifications import (notify_comment, notify_editor_judgment, notify_submitter_confirmation, notify_submitter_upload) -from indico.modules.events.editing.schemas import EditingConfirmationAction, EditingReviewAction +from indico.modules.events.editing.schemas import (EditableDumpSchema, EditingConfirmationAction, EditingFileTypeSchema, + EditingReviewAction) from indico.modules.events.logs import EventLogKind, EventLogRealm from indico.util.date_time import now_utc from indico.util.fs import secure_filename @@ -45,10 +44,10 @@ class InvalidEditableState(BadRequest): """ def __init__(self): - super(InvalidEditableState, self).__init__(_('The requested action is not possible on this revision')) + super().__init__(_('The requested action is not possible on this revision')) -def _ensure_latest_revision(revision): +def ensure_latest_revision(revision): if revision != revision.editable.revisions[-1]: raise InvalidEditableState @@ -69,7 +68,7 @@ def _make_editable_files(editable, files): return [] editable_files = [ EditingRevisionFile(file=file, file_type=file_type) - for file_type, file_list in files.viewitems() + for file_type, file_list in files.items() for file in file_list ] for ef in editable_files: @@ -99,7 +98,7 @@ def delete_editable(editable): def publish_editable_revision(revision): - _ensure_latest_revision(revision) + ensure_latest_revision(revision) revision.editable.published_revision = revision db.session.flush() logger.info('Revision %r marked as published', revision) @@ -107,7 +106,7 @@ def publish_editable_revision(revision): @no_autoflush def review_editable_revision(revision, editor, action, comment, tags, files=None): - _ensure_latest_revision(revision) + ensure_latest_revision(revision) _ensure_state(revision, initial=InitialRevisionState.ready_for_review, final=FinalRevisionState.none) revision.editor = editor revision.comment = comment @@ -121,9 +120,9 @@ def review_editable_revision(revision, editor, action, comment, tags, files=None }[action] db.session.flush() + new_revision = None if action == EditingReviewAction.accept: _ensure_publishable_files(revision) - publish_editable_revision(revision) elif action in (EditingReviewAction.update, EditingReviewAction.update_accept): final_state = FinalRevisionState.none editable_editor = None @@ -138,17 +137,15 @@ def review_editable_revision(revision, editor, action, comment, tags, files=None tags=revision.tags) _ensure_publishable_files(new_revision) revision.editable.revisions.append(new_revision) - if action == EditingReviewAction.update_accept: - db.session.flush() - publish_editable_revision(new_revision) db.session.flush() - notify_editor_judgment(revision, session.user) + notify_editor_judgment(revision, editor) logger.info('Revision %r reviewed by %s [%s]', revision, editor, action.name) + return new_revision @no_autoflush def confirm_editable_changes(revision, submitter, action, comment): - _ensure_latest_revision(revision) + ensure_latest_revision(revision) _ensure_state(revision, initial=InitialRevisionState.needs_submitter_confirmation, final=FinalRevisionState.none) revision.final_state = { EditingConfirmationAction.accept: FinalRevisionState.accepted, @@ -159,7 +156,6 @@ def confirm_editable_changes(revision, submitter, action, comment): db.session.flush() if action == EditingConfirmationAction.accept: _ensure_publishable_files(revision) - publish_editable_revision(revision) db.session.flush() notify_submitter_confirmation(revision, submitter, action) logger.info('Revision %r confirmed by %s [%s]', revision, submitter, action.name) @@ -167,14 +163,15 @@ def confirm_editable_changes(revision, submitter, action, comment): @no_autoflush def replace_revision(revision, user, comment, files, tags, initial_state=None): - _ensure_latest_revision(revision) + ensure_latest_revision(revision) _ensure_state(revision, initial=(InitialRevisionState.new, InitialRevisionState.ready_for_review), final=FinalRevisionState.none) revision.comment = comment revision.tags = tags revision.final_state = FinalRevisionState.replaced - new_revision = EditingRevision(submitter=user, + revision.editor = user + new_revision = EditingRevision(submitter=revision.submitter, initial_state=(initial_state or revision.initial_state), files=_make_editable_files(revision.editable, files)) revision.editable.revisions.append(new_revision) @@ -184,7 +181,7 @@ def replace_revision(revision, user, comment, files, tags, initial_state=None): @no_autoflush def create_submitter_revision(prev_revision, user, files): - _ensure_latest_revision(prev_revision) + ensure_latest_revision(prev_revision) _ensure_state(prev_revision, final=FinalRevisionState.needs_submitter_changes) new_revision = EditingRevision(submitter=user, initial_state=InitialRevisionState.ready_for_review, @@ -194,6 +191,7 @@ def create_submitter_revision(prev_revision, user, files): db.session.flush() notify_submitter_upload(new_revision) logger.info('Revision %r created by submitter %s', new_revision, user) + return new_revision def _ensure_latest_revision_with_final_state(revision): @@ -230,7 +228,7 @@ def undo_review(revision): @no_autoflush def create_revision_comment(revision, user, text, internal=False): - _ensure_latest_revision(revision) + ensure_latest_revision(revision) comment = EditingRevisionComment(user=user, text=text, internal=internal) revision.comments.append(comment) db.session.flush() @@ -240,7 +238,7 @@ def create_revision_comment(revision, user, text, internal=False): @no_autoflush def update_revision_comment(comment, updates): - _ensure_latest_revision(comment.revision) + ensure_latest_revision(comment.revision) comment.populate_from_dict(updates) comment.modified_dt = now_utc() db.session.flush() @@ -249,7 +247,7 @@ def update_revision_comment(comment, updates): @no_autoflush def delete_revision_comment(comment): - _ensure_latest_revision(comment.revision) + ensure_latest_revision(comment.revision) comment.is_deleted = True db.session.flush() logger.info('Comment on revision %r deleted: %r', comment.revision, comment) @@ -349,7 +347,7 @@ def unassign_editor(editable): db.session.flush() -def generate_editables_zip(editables): +def generate_editables_zip(event, editable_type, editables): buf = BytesIO() with ZipFile(buf, 'w', allowZip64=True) as zip_file: for editable in editables: @@ -362,21 +360,30 @@ def generate_editables_zip(editables): return send_file('files.zip', buf, 'application/zip', inline=False) +def generate_editables_json(event, editable_type, editables): + file_types = EditingFileType.query.with_parent(event).filter_by(type=editable_type).all() + file_types_dump = EditingFileTypeSchema(many=True).dump(file_types) + editables_dump = EditableDumpSchema(many=True).dump(editables) + response = jsonify(version=1, file_types=file_types_dump, editables=editables_dump) + response.headers['Content-Disposition'] = 'attachment; filename="editables.json"' + return response + + def _compose_filepath(editable, revision_file): file_obj = revision_file.file contrib = editable.contribution editable_type = editable.type.name - code = 'Editable-{}'.format(contrib.friendly_id) + code = f'Editable-{contrib.friendly_id}' if contrib.code: - code += '-{}'.format(contrib.code) + code += f'-{contrib.code}' - filepath = os.path.join(secure_filename('{}-{}'.format(contrib.title, contrib.id), - 'contribution-{}'.format(contrib.id)), + filepath = os.path.join(secure_filename(f'{contrib.title}-{contrib.id}', + f'contribution-{contrib.id}'), editable_type, code, revision_file.file_type.name) filename, ext = os.path.splitext(file_obj.filename) filename = secure_filename(file_obj.filename, - 'revision-file-{}-{}{}'.format(revision_file.revision_id, file_obj.id, ext)) + f'revision-file-{revision_file.revision_id}-{file_obj.id}{ext}') return os.path.join(filepath, filename) diff --git a/indico/modules/events/editing/operations_test.py b/indico/modules/events/editing/operations_test.py index 0492c1514c7..abdad80dad9 100644 --- a/indico/modules/events/editing/operations_test.py +++ b/indico/modules/events/editing/operations_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import pytest from indico.modules.events.editing.models.editable import Editable, EditableType diff --git a/indico/modules/events/editing/schemas.py b/indico/modules/events/editing/schemas.py index f55863277fe..598f192a506 100644 --- a/indico/modules/events/editing/schemas.py +++ b/indico/modules/events/editing/schemas.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import re from markupsafe import escape @@ -15,6 +13,7 @@ from sqlalchemy import func from indico.core.marshmallow import mm +from indico.modules.categories.models.roles import CategoryRole from indico.modules.events.contributions.models.contributions import Contribution from indico.modules.events.contributions.schemas import ContributionSchema from indico.modules.events.editing.models.comments import EditingRevisionComment @@ -24,16 +23,27 @@ from indico.modules.events.editing.models.revision_files import EditingRevisionFile from indico.modules.events.editing.models.revisions import EditingRevision, InitialRevisionState from indico.modules.events.editing.models.tags import EditingTag +from indico.modules.events.util import get_all_user_roles from indico.modules.users import User from indico.util.caching import memoize_request +from indico.util.enum import IndicoEnum from indico.util.i18n import _ from indico.util.marshmallow import PrincipalList, not_empty from indico.util.string import natural_sort_key -from indico.util.struct.enum import IndicoEnum from indico.web.flask.util import url_for from indico.web.forms.colors import get_sui_colors +def _get_anonymous_user(): + return { + 'identifier': 'AnonymousUser', + 'avatar_url': url_for('assets.avatar'), + 'id': -1, + 'full_name': 'Someone', + 'anonymous': True, + } + + class RevisionStateSchema(mm.Schema): title = fields.String() name = fields.String() @@ -46,13 +56,13 @@ class EditableStateSchema(mm.Schema): css_class = fields.String() -class EditingUserSchema(mm.ModelSchema): +class EditingUserSchema(mm.SQLAlchemyAutoSchema): class Meta: model = User - fields = ('id', 'avatar_bg_color', 'full_name', 'identifier') + fields = ('id', 'full_name', 'identifier', 'avatar_url') -class EditingFileTypeSchema(mm.ModelSchema): +class EditingFileTypeSchema(mm.SQLAlchemyAutoSchema): class Meta: model = EditingFileType fields = ('id', 'name', 'extensions', 'allow_multiple_files', 'required', 'publishable', 'is_used', @@ -72,7 +82,7 @@ def sort_list(self, data, many, **kwargs): return data -class EditingTagSchema(mm.ModelSchema): +class EditingTagSchema(mm.SQLAlchemyAutoSchema): class Meta: model = EditingTag fields = ( @@ -91,7 +101,7 @@ def sort_list(self, data, many, **kwargs): return data -class EditingRevisionFileSchema(mm.ModelSchema): +class EditingRevisionFileSchema(mm.SQLAlchemyAutoSchema): class Meta: model = EditingRevisionFile fields = ('uuid', 'filename', 'size', 'content_type', 'file_type', 'download_url', 'external_download_url') @@ -104,7 +114,14 @@ class Meta: external_download_url = fields.String() -class EditingRevisionCommentSchema(mm.ModelSchema): +class EditingRevisionSignedFileSchema(EditingRevisionFileSchema): + class Meta(EditingRevisionFileSchema.Meta): + fields = ('uuid', 'filename', 'size', 'content_type', 'file_type', 'signed_download_url') + + signed_download_url = fields.String(attribute='file.signed_download_url') + + +class EditingRevisionCommentSchema(mm.SQLAlchemyAutoSchema): class Meta: model = EditingRevisionComment fields = ('id', 'user', 'created_dt', 'modified_dt', 'internal', 'system', 'text', 'html', 'can_modify', @@ -116,13 +133,20 @@ class Meta: can_modify = fields.Function(lambda comment, ctx: comment.can_modify(ctx.get('user'))) modify_comment_url = fields.Function(lambda comment: url_for('event_editing.api_edit_comment', comment)) + @post_dump(pass_original=True) + def anonymize_user(self, data, orig, **kwargs): + can_see_editor_names_fn = self.context.get('can_see_editor_names') + if can_see_editor_names_fn and data['user'] and not can_see_editor_names_fn(self.context['user'], orig.user): + data['user'] = _get_anonymous_user() + return data + -class EditingRevisionSchema(mm.ModelSchema): +class EditingRevisionSchema(mm.SQLAlchemyAutoSchema): class Meta: model = EditingRevision fields = ('id', 'created_dt', 'submitter', 'editor', 'files', 'comment', 'comment_html', 'comments', - 'initial_state', 'final_state', 'tags', 'create_comment_url', 'download_files_url', 'review_url', - 'confirm_url') + 'initial_state', 'final_state', 'tags', 'create_comment_url', 'download_files_url', + 'review_url', 'confirm_url', 'custom_actions', 'custom_action_url') comment_html = fields.Function(lambda rev: escape(rev.comment)) submitter = fields.Nested(EditingUserSchema) @@ -136,6 +160,8 @@ class Meta: download_files_url = fields.Function(lambda revision: url_for('event_editing.revision_files_export', revision)) review_url = fields.Function(lambda revision: url_for('event_editing.api_review_editable', revision)) confirm_url = fields.Method('_get_confirm_url') + custom_action_url = fields.Function(lambda revision: url_for('event_editing.api_custom_action', revision)) + custom_actions = fields.Function(lambda revision, ctx: ctx.get('custom_actions', {}).get(revision, [])) def _get_confirm_url(self, revision): if revision.initial_state == InitialRevisionState.needs_submitter_confirmation and not revision.final_state: @@ -151,8 +177,22 @@ def sort_tags(self, data, **kwargs): data['tags'].sort(key=lambda tag: natural_sort_key(tag['verbose_title'])) return data + @post_dump(pass_original=True) + def anonymize_users(self, data, orig, **kwargs): + can_see_editor_names_fn = self.context.get('can_see_editor_names') + if can_see_editor_names_fn: + if data['editor'] and not can_see_editor_names_fn(self.context['user'], orig.editor): + data['editor'] = _get_anonymous_user() + if data['submitter'] and not can_see_editor_names_fn(self.context['user'], orig.submitter): + data['submitter'] = _get_anonymous_user() + return data + + +class EditingRevisionSignedSchema(EditingRevisionSchema): + files = fields.List(fields.Nested(EditingRevisionSignedFileSchema)) + -class EditableSchema(mm.ModelSchema): +class EditableSchema(mm.SQLAlchemyAutoSchema): class Meta: model = Editable fields = ('id', 'type', 'editor', 'revisions', 'contribution', 'can_comment', 'review_conditions_valid', @@ -177,8 +217,20 @@ class Meta: editing_enabled = fields.Boolean() state = fields.Nested(EditableStateSchema) + @post_dump(pass_original=True) + def anonymize_editor(self, data, orig, **kwargs): + can_see_editor_names_fn = self.context.get('can_see_editor_names') + if can_see_editor_names_fn and data['editor'] and not can_see_editor_names_fn(self.context['user']): + data['editor'] = _get_anonymous_user() + return data + -class EditableBasicSchema(mm.ModelSchema): +class EditableDumpSchema(EditableSchema): + class Meta(EditableSchema.Meta): + fields = [f for f in EditableSchema.Meta.fields if not f.startswith('can_')] + + +class EditableBasicSchema(mm.SQLAlchemyAutoSchema): class Meta: model = Editable fields = ('id', 'type', 'state', 'editor', 'timeline_url', 'revision_count') @@ -188,7 +240,7 @@ class Meta: timeline_url = fields.String() -class EditingEditableListSchema(mm.ModelSchema): +class EditingEditableListSchema(mm.SQLAlchemyAutoSchema): class Meta: model = Contribution fields = ('id', 'friendly_id', 'title', 'code', 'editable') @@ -203,7 +255,7 @@ def _get_editable(self, contribution): return EditableBasicSchema().dump(editable) -class FilteredEditableSchema(mm.ModelSchema): +class FilteredEditableSchema(mm.SQLAlchemyAutoSchema): class Meta: model = Editable fields = ('contribution_id', 'contribution_title', 'contribution_code', 'contribution_friendly_id', @@ -240,7 +292,7 @@ class ReviewEditableArgs(mm.Schema): comment = fields.String(missing='') @validates_schema(skip_on_field_errors=True) - def validate_everything(self, data): + def validate_everything(self, data, **kwargs): if data['action'] != EditingReviewAction.accept and not data['comment']: raise ValidationError('This field is required', 'comment') @@ -260,7 +312,7 @@ class Meta: system = fields.Bool(missing=False) @validates('code') - def _check_for_unique_tag_code(self, code): + def _check_for_unique_tag_code(self, code, **kwargs): event = self.context['event'] tag = self.context['tag'] query = EditingTag.query.with_parent(event).filter(func.lower(EditingTag.code) == code.lower()) @@ -270,7 +322,7 @@ def _check_for_unique_tag_code(self, code): raise ValidationError(_('Tag code must be unique')) @validates('system') - def _check_only_services_set_system_tags(self, value): + def _check_only_services_set_system_tags(self, value, **kwargs): if value and not self.context['is_service_call']: raise ValidationError('Only custom editing workflows can set system tags') @@ -287,7 +339,7 @@ class Meta: publishable = fields.Boolean() @validates('name') - def _check_for_unique_file_type_name(self, name): + def _check_for_unique_file_type_name(self, name, **kwargs): event = self.context['event'] file_type = self.context['file_type'] editable_type = self.context['editable_type'] @@ -300,18 +352,18 @@ def _check_for_unique_file_type_name(self, name): raise ValidationError(_('Name must be unique')) @validates('filename_template') - def _check_for_correct_filename_template(self, template): + def _check_for_correct_filename_template(self, template, **kwargs): if template is not None and '.' in template: raise ValidationError(_('Filename template cannot include dots')) @validates('extensions') - def _check_for_correct_extensions_format(self, extensions): + def _check_for_correct_extensions_format(self, extensions, **kwargs): for extension in extensions: if re.match(r'^[*.]+', extension): raise ValidationError(_('Extensions cannot have leading dots')) @validates('publishable') - def _check_if_can_unset_or_delete(self, publishable): + def _check_if_can_unset_or_delete(self, publishable, **kwargs): event = self.context['event'] file_type = self.context['file_type'] editable_type = self.context['editable_type'] @@ -333,7 +385,7 @@ class Meta: file_types = fields.List(fields.Int(), required=True, validate=not_empty) @validates('file_types') - def _validate_file_types(self, file_types): + def _validate_file_types(self, file_types, **kwargs): editable_type = self.context['editable_type'] event = self.context['event'] event_file_types = {ft.id for ft in event.editing_file_types} @@ -363,3 +415,56 @@ class Meta: rh_context = ('event',) principals = PrincipalList(many=True, allow_event_roles=True, allow_category_roles=True) + + +class ReviewCommentSchema(mm.Schema): + text = fields.String(required=True) + internal = fields.Boolean(missing=False) + + +class RoleSchema(mm.Schema): + name = fields.String() + code = fields.String() + source = fields.Method('_get_source') + + def _get_source(self, role): + return 'category' if isinstance(role, CategoryRole) else 'event' + + +class ServiceUserSchema(mm.SQLAlchemyAutoSchema): + class Meta: + model = User + fields = ('id', 'full_name', 'email', 'roles', 'manager', 'submitter', 'editor') + + roles = fields.Method('_get_roles') + manager = fields.Function(lambda user, ctx: ctx['editable'].event.can_manage(user, 'editing_manager')) + submitter = fields.Function(lambda user, ctx: ctx['editable'].can_perform_submitter_actions(user)) + editor = fields.Function(lambda user, ctx: ctx['editable'].can_perform_editor_actions(user)) + + def _get_roles(self, user): + event = self.context['editable'].event + event_roles, category_roles = get_all_user_roles(event, user) + roles = RoleSchema(many=True).dump(event_roles | category_roles) + return sorted(roles, key=lambda r: (r['source'], r['code'])) + + +class ServiceReviewEditableSchema(mm.Schema): + publish = fields.Boolean(missing=True) + comment = fields.String() + comments = fields.List(fields.Nested(ReviewCommentSchema)) + tags = fields.List(fields.Int()) + + +class ServiceActionSchema(mm.Schema): + name = fields.String(required=True) + title = fields.String(required=True) + color = fields.String(missing=None) + icon = fields.String(missing=None) + confirm = fields.String(missing=None) + + +class ServiceActionResultSchema(mm.Schema): + publish = fields.Boolean() + comments = fields.List(fields.Nested(ReviewCommentSchema)) + tags = fields.List(fields.Int()) + redirect = fields.String() diff --git a/indico/modules/events/editing/service.py b/indico/modules/events/editing/service.py index de0915736e5..466841852c7 100644 --- a/indico/modules/events/editing/service.py +++ b/indico/modules/events/editing/service.py @@ -1,21 +1,26 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import requests +from marshmallow import ValidationError from werkzeug.urls import url_parse import indico from indico.core.config import config +from indico.core.db import db from indico.modules.events.editing import logger from indico.modules.events.editing.models.editable import EditableType -from indico.modules.events.editing.schemas import EditingRevisionFileSchema +from indico.modules.events.editing.models.revisions import FinalRevisionState +from indico.modules.events.editing.operations import create_revision_comment, publish_editable_revision +from indico.modules.events.editing.schemas import (EditableBasicSchema, EditingRevisionSignedSchema, + ServiceActionResultSchema, ServiceActionSchema, + ServiceReviewEditableSchema, ServiceUserSchema) from indico.modules.events.editing.settings import editing_settings +from indico.modules.users import User from indico.util.caching import memoize_redis from indico.util.i18n import _ from indico.web.flask.util import url_for @@ -30,7 +35,7 @@ def __init__(self, exc): except (ValueError, KeyError): # not json or not error field error = None - super(ServiceRequestFailed, self).__init__(error or unicode(exc)) + super().__init__(error or str(exc)) @memoize_redis(30) @@ -39,12 +44,12 @@ def check_service_url(url): resp = requests.get(url + '/info', allow_redirects=False) resp.raise_for_status() if resp.status_code != 200: - raise requests.HTTPError('Unexpected status code: {}'.format(resp.status_code), response=resp) + raise requests.HTTPError(f'Unexpected status code: {resp.status_code}', response=resp) info = resp.json() - except requests.ConnectionError as exc: + except requests.ConnectionError: return {'info': None, 'error': _('Connection failed')} except requests.RequestException as exc: - return {'info': None, 'error': unicode(ServiceRequestFailed(exc))} + return {'info': None, 'error': str(ServiceRequestFailed(exc))} if not all(x in info for x in ('name', 'version')): return {'error': _('Invalid response')} return {'error': None, 'info': info} @@ -56,12 +61,25 @@ def _build_url(event, path): def _get_headers(event, include_token=True): headers = {'Accept': 'application/json', - 'User-Agent': 'Indico/{}'.format(indico.__version__)} + 'User-Agent': f'Indico/{indico.__version__}'} if include_token: headers['Authorization'] = 'Bearer {}'.format(editing_settings.get(event, 'service_token')) return headers +def _log_service_error(exc, msg): + payload = None + if exc.response is not None: + try: + payload = exc.response.json() + except ValueError: + pass + if payload is not None: + logger.exception(f'{msg}: %s', payload) + else: + logger.exception(msg) + + def make_event_identifier(event): data = url_parse(config.BASE_URL) parts = data.netloc.split('.') @@ -81,61 +99,46 @@ def service_handle_enabled(event): 'title': event.title, 'url': event.external_url, 'token': editing_settings.get(event, 'service_token'), - 'endpoints': { - 'tags': { - 'create': url_for('.api_create_tag', event, _external=True), - 'list': url_for('.api_tags', event, _external=True) - }, - 'editable_types': url_for('.api_enabled_editable_types', event, _external=True), - 'file_types': { - t.name: { - 'create': url_for('.api_add_file_type', event, type=t.name, _external=True), - 'list': url_for('.api_file_types', event, type=t.name, _external=True), - } for t in EditableType - } - } + 'endpoints': _get_event_endpoints(event) } try: - resp = requests.put(_build_url(event, '/event/{}'.format(_get_event_identifier(event))), + resp = requests.put(_build_url(event, f'/event/{_get_event_identifier(event)}'), headers=_get_headers(event, include_token=False), json=data) resp.raise_for_status() except requests.RequestException as exc: - logger.exception('Registering event with service failed') + _log_service_error(exc, 'Registering event with service failed') raise ServiceRequestFailed(exc) def service_handle_disconnected(event): try: - resp = requests.delete(_build_url(event, '/event/{}'.format(_get_event_identifier(event))), + resp = requests.delete(_build_url(event, f'/event/{_get_event_identifier(event)}'), headers=_get_headers(event)) resp.raise_for_status() except requests.RequestException as exc: - logger.exception('Disconnecting event from service failed') + _log_service_error(exc, 'Disconnecting event from service failed') raise ServiceRequestFailed(exc) def service_get_status(event): try: - resp = requests.get(_build_url(event, '/event/{}'.format(_get_event_identifier(event))), + resp = requests.get(_build_url(event, f'/event/{_get_event_identifier(event)}'), headers=_get_headers(event)) resp.raise_for_status() - except requests.ConnectionError as exc: + except requests.ConnectionError: return {'status': None, 'error': _('Connection failed')} except requests.RequestException as exc: - return {'status': None, 'error': unicode(ServiceRequestFailed(exc))} + return {'status': None, 'error': str(ServiceRequestFailed(exc))} return {'status': resp.json(), 'error': None} -def service_handle_new_editable(editable): +def service_handle_new_editable(editable, user): revision = editable.revisions[-1] data = { - 'files': EditingRevisionFileSchema().dump(revision.files, many=True), - 'endpoints': { - 'revisions': { - 'replace': url_for('.api_replace_revision', revision, _external=True) - }, - 'file_upload': url_for('.api_upload', editable, _external=True) - } + 'editable': EditableBasicSchema().dump(editable), + 'revision': EditingRevisionSignedSchema().dump(revision), + 'endpoints': _get_revision_endpoints(revision), + 'user': ServiceUserSchema(context={'editable': editable}).dump(user), } try: path = '/event/{}/editable/{}/{}'.format( @@ -146,5 +149,124 @@ def service_handle_new_editable(editable): resp = requests.put(_build_url(editable.event, path), headers=_get_headers(editable.event), json=data) resp.raise_for_status() except requests.RequestException as exc: - logger.exception('Failed calling listener for editable') + _log_service_error(exc, 'Calling listener for new editable failed') raise ServiceRequestFailed(exc) + + +def service_handle_review_editable(editable, user, action, parent_revision, revision=None): + new_revision = revision or parent_revision + data = { + 'action': action.name, + 'revision': EditingRevisionSignedSchema().dump(new_revision), + 'endpoints': _get_revision_endpoints(new_revision), + 'user': ServiceUserSchema(context={'editable': editable}).dump(user), + } + try: + path = '/event/{}/editable/{}/{}/{}'.format( + _get_event_identifier(editable.event), + editable.type.name, + editable.contribution_id, + new_revision.id + ) + resp = requests.post(_build_url(editable.event, path), headers=_get_headers(editable.event), + json=data) + resp.raise_for_status() + resp = ServiceReviewEditableSchema().load(resp.json()) + + if 'comment' in resp: + parent_revision.comment = resp['comment'] + if 'tags' in resp: + resp_tag_ids = set(map(int, resp['tags'])) + parent_revision.tags = {tag for tag in editable.event.editing_tags if tag.id in resp_tag_ids} + for comment in resp.get('comments', []): + create_revision_comment(new_revision, User.get_system_user(), comment['text'], internal=comment['internal']) + + db.session.flush() + return resp + except (requests.RequestException, ValidationError) as exc: + _log_service_error(exc, 'Calling listener for editable revision failed') + raise ServiceRequestFailed(exc) + + +def service_get_custom_actions(editable, revision, user): + data = { + 'revision': EditingRevisionSignedSchema().dump(revision), + 'user': ServiceUserSchema(context={'editable': editable}).dump(user), + } + + path = '/event/{}/editable/{}/{}/{}/actions'.format( + _get_event_identifier(editable.event), + editable.type.name, + editable.contribution_id, + revision.id + ) + try: + resp = requests.post(_build_url(editable.event, path), headers=_get_headers(editable.event), json=data) + resp.raise_for_status() + return ServiceActionSchema(many=True).load(resp.json()) + except (requests.RequestException, ValidationError) as exc: + _log_service_error(exc, 'Calling listener for custom actions failed') + raise ServiceRequestFailed(exc) + + +def service_handle_custom_action(editable, revision, user, action): + data = { + 'action': action, + 'revision': EditingRevisionSignedSchema().dump(revision), + 'endpoints': _get_revision_endpoints(revision), + 'user': ServiceUserSchema(context={'editable': editable}).dump(user), + } + try: + path = '/event/{}/editable/{}/{}/{}/action'.format( + _get_event_identifier(editable.event), + editable.type.name, + editable.contribution_id, + revision.id + ) + resp = requests.post(_build_url(editable.event, path), headers=_get_headers(editable.event), json=data) + resp.raise_for_status() + resp = ServiceActionResultSchema().load(resp.json()) + except (requests.RequestException, ValidationError) as exc: + _log_service_error(exc, 'Calling listener for triggering custom action failed') + raise ServiceRequestFailed(exc) + + if revision.final_state == FinalRevisionState.accepted: + publish = resp.get('publish') + if publish: + publish_editable_revision(revision) + elif publish is False: + revision.editable.published_revision = None + if 'tags' in resp: + resp_tag_ids = set(map(int, resp['tags'])) + revision.tags = {tag for tag in editable.event.editing_tags if tag.id in resp_tag_ids} + for comment in resp.get('comments', []): + create_revision_comment(revision, User.get_system_user(), comment['text'], internal=comment['internal']) + db.session.flush() + return resp + + +def _get_event_endpoints(event): + return { + 'tags': { + 'create': url_for('.api_create_tag', event, _external=True), + 'list': url_for('.api_tags', event, _external=True) + }, + 'editable_types': url_for('.api_enabled_editable_types', event, _external=True), + 'file_types': { + t.name: { + 'create': url_for('.api_add_file_type', event, type=t.name, _external=True), + 'list': url_for('.api_file_types', event, type=t.name, _external=True), + } for t in EditableType + } + } + + +def _get_revision_endpoints(revision): + return { + 'revisions': { + 'details': url_for('.api_editable', revision, _external=True), + 'replace': url_for('.api_replace_revision', revision, _external=True), + 'undo': url_for('.api_undo_review', revision, _external=True) + }, + 'file_upload': url_for('.api_upload', revision, _external=True) + } diff --git a/indico/modules/events/editing/settings.py b/indico/modules/events/editing/settings.py index 3007a892132..34938dc39f2 100644 --- a/indico/modules/events/editing/settings.py +++ b/indico/modules/events/editing/settings.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.editing.models.editable import EditableType from indico.modules.events.settings import EventSettingsProxy @@ -20,6 +18,7 @@ _defaults = { 'self_assign_allowed': False, + 'anonymous_team': False, 'submission_enabled': False, 'editing_enabled': False, } diff --git a/indico/modules/events/editing/templates/emails/comment_notification.txt b/indico/modules/events/editing/templates/emails/comment_notification.txt index 2fa2818b598..1957a3da180 100644 --- a/indico/modules/events/editing/templates/emails/comment_notification.txt +++ b/indico/modules/events/editing/templates/emails/comment_notification.txt @@ -3,5 +3,5 @@ {% block subject -%}New comment{%- endblock %} {% block body_text -%} -{{ author_name }} has posted a new comment. +{{ author_name or 'Someone' }} has posted a new comment. {% endblock %} diff --git a/indico/modules/events/editing/templates/emails/editor_judgment_notification.txt b/indico/modules/events/editing/templates/emails/editor_judgment_notification.txt index 998d54c1fb8..5b26c26ce21 100644 --- a/indico/modules/events/editing/templates/emails/editor_judgment_notification.txt +++ b/indico/modules/events/editing/templates/emails/editor_judgment_notification.txt @@ -3,5 +3,5 @@ {% block subject -%}New editor judgment{%- endblock %} {% block body_text -%} -{{ editor_name }} has left a judgment to an editable you are submitter of. +{{ editor_name or 'Someone' }} has left a judgment to an editable you are submitter of. {% endblock %} diff --git a/indico/modules/events/editing/util.py b/indico/modules/events/editing/util.py index 07953e1b538..18c580aecc8 100644 --- a/indico/modules/events/editing/util.py +++ b/indico/modules/events/editing/util.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/editing/views.py b/indico/modules/events/editing/views.py index 48ecf4cd955..ef1625f5598 100644 --- a/indico/modules/events/editing/views.py +++ b/indico/modules/events/editing/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement diff --git a/indico/modules/events/export.py b/indico/modules/events/export.py index 2f82d9281f7..b0ce4ac5c42 100644 --- a/indico/modules/events/export.py +++ b/indico/modules/events/export.py @@ -1,17 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import posixpath import re import tarfile -from collections import OrderedDict, defaultdict +from collections import defaultdict from datetime import date, datetime from io import BytesIO from operator import itemgetter @@ -28,6 +26,7 @@ from indico.core.config import config from indico.core.db import db from indico.core.db.sqlalchemy.principals import PrincipalType +from indico.core.db.sqlalchemy.util.models import get_all_models from indico.core.storage.backend import get_storage from indico.modules.events import Event, EventLogKind, EventLogRealm from indico.modules.events.contributions import Contribution @@ -41,7 +40,7 @@ from indico.modules.users.util import get_user_by_email from indico.util.console import cformat from indico.util.date_time import now_utc -from indico.util.string import strict_unicode +from indico.util.string import strict_str _notset = object() @@ -76,7 +75,7 @@ def import_event(source_file, category_id=0, create_users=None, verbose=False, f def _model_to_table(name): - """Resolve a model name to a full table name (unless it's already one)""" + """Resolve a model name to a full table name (unless it's already one).""" return getattr(db.m, name).__table__.fullname if name[0].isupper() else name @@ -85,8 +84,7 @@ def _make_globals(**extra): Build a globals dict for the exec/eval environment that contains all the models and whatever extra data is needed. """ - globals_ = {name: cls for name, cls in db.Model._decl_class_registry.iteritems() - if hasattr(cls, '__table__')} + globals_ = {cls.__name__: cls for cls in get_all_models() if hasattr(cls, '__table__')} globals_.update(extra) return globals_ @@ -95,8 +93,8 @@ def _exec_custom(code, **extra): """Execute a custom code snippet and return all non-underscored values.""" globals_ = _make_globals(**extra) locals_ = {} - exec code in globals_, locals_ - return {unicode(k): v for k, v in locals_.iteritems() if k[0] != '_'} + exec(code, globals_, locals_) + return {str(k): v for k, v in locals_.items() if k[0] != '_'} def _resolve_col(col): @@ -105,7 +103,7 @@ def _resolve_col(col): :param col: A string containing a Python expression, a model attribute or a Column instance. """ - attr = eval(col, _make_globals()) if isinstance(col, basestring) else col + attr = eval(col, _make_globals()) if isinstance(col, str) else col if isinstance(attr, db.Column): return attr assert len(attr.prop.columns) == 1 @@ -122,23 +120,23 @@ def _get_single_fk(col): def _get_pk(table): """Get the single column that is the table's PK.""" - pks = inspect(table).primary_key.columns.values() + pks = list(inspect(table).primary_key.columns.values()) assert len(pks) == 1 return pks[0] def _has_single_pk(table): """Check if the table has a single PK.""" - return len(inspect(table).primary_key.columns.values()) == 1 + return len(list(inspect(table).primary_key.columns.values())) == 1 def _get_inserted_pk(result): - """Get the single PK value inserted by a query""" + """Get the single PK value inserted by a query.""" assert len(result.inserted_primary_key) == 1 return result.inserted_primary_key[0] -class EventExporter(object): +class EventExporter: def __init__(self, event, target_file): self.event = event self.target_file = target_file @@ -151,8 +149,10 @@ def __init__(self, event, target_file): self.users = {} def _add_file(self, name, size, data): - if isinstance(data, basestring): + if isinstance(data, bytes): data = BytesIO(data) + elif isinstance(data, str): + data = BytesIO(data.encode()) info = tarfile.TarInfo(name) info.size = size self.archive.addfile(info, data) @@ -175,30 +175,30 @@ def _process_tablespec(tablename, tablespec): tablespec.setdefault('skipif', None) tablespec.setdefault('order', None) tablespec.setdefault('allow_duplicates', False) - fks = OrderedDict() + fks = {} for fk_name in tablespec['fks']: col = _resolve_col(fk_name) fk = _get_single_fk(col) fks.setdefault(fk.column.name, []).append(col) tablespec['fks'] = fks - tablespec['fks_out'] = OrderedDict((fk, _get_single_fk(db.metadata.tables[tablename].c[fk]).column) - for fk in tablespec['fks_out']) + tablespec['fks_out'] = {fk: _get_single_fk(db.metadata.tables[tablename].c[fk]).column + for fk in tablespec['fks_out']} return tablespec with open(os.path.join(current_app.root_path, 'modules', 'events', 'export.yaml')) as f: spec = yaml.safe_load(f) - return {_model_to_table(k): _process_tablespec(_model_to_table(k), v) for k, v in spec['export'].iteritems()} + return {_model_to_table(k): _process_tablespec(_model_to_table(k), v) for k, v in spec['export'].items()} def _get_reverse_fk_map(self): - """Build a mapping between columns and incoming FKs""" + """Build a mapping between columns and incoming FKs.""" legacy_tables = {'events.legacy_contribution_id_map', 'events.legacy_subcontribution_id_map', 'attachments.legacy_attachment_id_map', 'event_registration.legacy_registration_map', 'events.legacy_session_block_id_map', 'events.legacy_image_id_map', 'events.legacy_session_id_map', 'events.legacy_page_id_map', 'categories.legacy_id_map', 'events.legacy_id_map', 'attachments.legacy_folder_id_map'} fk_targets = defaultdict(set) - for name, table in db.metadata.tables.iteritems(): + for name, table in db.metadata.tables.items(): if name in legacy_tables: continue for column in table.columns: @@ -207,7 +207,7 @@ def _get_reverse_fk_map(self): return dict(fk_targets) def _get_uuid(self): - uuid = unicode(uuid4()) + uuid = str(uuid4()) if uuid in self.used_uuids: # VERY unlikely but just in case... return self._get_uuid() @@ -236,11 +236,11 @@ def _make_idref(self, column, value, incoming=False, target_column=None): if incoming: assert column.primary_key assert target_column is None - fullname = '{}.{}'.format(column.table.fullname, column.name) + fullname = f'{column.table.fullname}.{column.name}' type_ = 'idref_set' else: if target_column is not None: - fullname = '{}.{}'.format(target_column.table.fullname, target_column.name) + fullname = f'{target_column.table.fullname}.{target_column.name}' else: fk = _get_single_fk(column) fullname = fk.target_fullname @@ -321,7 +321,7 @@ def _serialize_objects(self, table, filter_): if spec['skipif'] and eval(spec['skipif'], _make_globals(ROW=row)): continue rowdict = row._asdict() - pk = tuple(v for k, v in rowdict.viewitems() if table.c[k].primary_key) + pk = tuple(v for k, v in rowdict.items() if table.c[k].primary_key) if (table.fullname, pk) in self.seen_rows: if spec['allow_duplicates']: continue @@ -329,9 +329,9 @@ def _serialize_objects(self, table, filter_): raise Exception('Trying to serialize already-serialized row') self.seen_rows.add((table.fullname, pk)) data = {} - for col, value in rowdict.viewitems(): - col = unicode(col) # col names are `quoted_name` objects - col_fullname = '{}.{}'.format(table.fullname, col) + for col, value in rowdict.items(): + col = str(col) # col names are `quoted_name` objects + col_fullname = f'{table.fullname}.{col}' col_custom = spec['cols'].get(col, _notset) colspec = table.c[col] if col_custom is None: @@ -341,7 +341,7 @@ def _serialize_objects(self, table, filter_): # column has custom code to process its value (and possibly name) if value is not None: def _get_event_idref(): - key = '{}.{}'.format(Event.__table__.fullname, Event.id.name) + key = f'{Event.__table__.fullname}.{Event.id.name}' assert key in self.id_map return 'idref', self.id_map[key][self.event.id] @@ -365,23 +365,21 @@ def _make_id_ref(target, id_): self._process_file(data) # export objects referenced in outgoing FKs before the row # itself as the FK column might not be nullable - for col, fk in spec['fks_out'].iteritems(): + for col, fk in spec['fks_out'].items(): value = rowdict[col] - for x in self._serialize_objects(fk.table, value == fk): - yield x + yield from self._serialize_objects(fk.table, value == fk) yield table.fullname, data # serialize objects referencing the current row, but don't export them yet - for col, fks in spec['fks'].iteritems(): + for col, fks in spec['fks'].items(): value = rowdict[col] cascaded += [x for fk in fks for x in self._serialize_objects(fk.table, value == fk)] # we only add incoming fks after being done with all objects in case one # of the referenced objects references another object from the current table # that has not been serialized yet (e.g. abstract reviews proposing as duplicate) - for x in cascaded: - yield x + yield from cascaded -class EventImporter(object): +class EventImporter: def __init__(self, source_file, category_id=0, create_users=None, verbose=False, force=False): self.source_file = source_file self.category_id = category_id @@ -400,7 +398,7 @@ def __init__(self, source_file, category_id=0, create_users=None, verbose=False, def _load_spec(self): def _resolve_col_name(col): colspec = _resolve_col(col) - return '{}.{}'.format(colspec.table.fullname, colspec.name) + return f'{colspec.table.fullname}.{colspec.name}' def _process_format(fmt, _re=re.compile(r'<([^>]+)>')): fmt = _re.sub(r'%{reset}%{cyan}\1%{reset}%{blue!}', fmt) @@ -410,17 +408,17 @@ def _process_format(fmt, _re=re.compile(r'<([^>]+)>')): spec = yaml.safe_load(f) spec = spec['import'] - spec['defaults'] = {_model_to_table(k): v for k, v in spec.get('defaults', {}).iteritems()} - spec['custom'] = {_model_to_table(k): v for k, v in spec.get('custom', {}).iteritems()} - spec['missing_users'] = {_resolve_col_name(k): v for k, v in spec.get('missing_users', {}).iteritems()} - spec['verbose'] = {_model_to_table(k): _process_format(v) for k, v in spec.get('verbose', {}).iteritems()} + spec['defaults'] = {_model_to_table(k): v for k, v in spec.get('defaults', {}).items()} + spec['custom'] = {_model_to_table(k): v for k, v in spec.get('custom', {}).items()} + spec['missing_users'] = {_resolve_col_name(k): v for k, v in spec.get('missing_users', {}).items()} + spec['verbose'] = {_model_to_table(k): _process_format(v) for k, v in spec.get('verbose', {}).items()} return spec def _load_users(self, data): if not data['users']: return missing = {} - for uuid, userdata in data['users'].iteritems(): + for uuid, userdata in data['users'].items(): if userdata is None: self.user_map[uuid] = self.system_user_id continue @@ -435,7 +433,7 @@ def _load_users(self, data): if missing: click.secho('The following users from the import data could not be mapped to existing users:', fg='yellow') table_data = [['First Name', 'Last Name', 'Email', 'Affiliation']] - for userdata in sorted(missing.itervalues(), key=itemgetter('first_name', 'last_name', 'email')): + for userdata in sorted(missing.values(), key=itemgetter('first_name', 'last_name', 'email')): table_data.append([userdata['first_name'], userdata['last_name'], userdata['email'], userdata['affiliation']]) table = AsciiTable(table_data) @@ -453,7 +451,7 @@ def _load_users(self, data): create_users = self.create_users if create_users: click.secho('Creating missing users', fg='magenta') - for uuid, userdata in missing.iteritems(): + for uuid, userdata in missing.items(): user = User(first_name=userdata['first_name'], last_name=userdata['last_name'], email=userdata['email'], @@ -489,10 +487,10 @@ def deserialize(self): # of circular dependencies where one of the IDs is not available # when the row is inserted). click.secho('BUG: Not all deferred idrefs have been consumed', fg='red') - for uuid, values in self.deferred_idrefs.iteritems(): - click.secho('{}:'.format(uuid), fg='yellow', bold=True) + for uuid, values in self.deferred_idrefs.items(): + click.secho(f'{uuid}:', fg='yellow', bold=True) for table, col, pk_value in values: - click.secho(' - {}.{} ({})'.format(table.fullname, col, pk_value), fg='yellow') + click.secho(f' - {table.fullname}.{col} ({pk_value})', fg='yellow') raise Exception('Not all deferred idrefs have been consumed') event = Event.get(self.event_id) event.log(EventLogRealm.event, EventLogKind.other, 'Event', 'Event imported from another Indico instance') @@ -554,7 +552,7 @@ def _convert_value(self, colspec, value): try: return self.user_map[value] except KeyError: - mode = self.spec['missing_users']['{}.{}'.format(colspec.table.fullname, colspec.name)] + mode = self.spec['missing_users'][f'{colspec.table.fullname}.{colspec.name}'] if mode == 'system': return self.system_user_id elif mode == 'none': @@ -570,8 +568,8 @@ def _get_file_storage_path(self, id_, filename): # we use a generic path to store all imported files since we # are on the table level here and thus cannot use relationships # and the orignal models' logic to construct paths - path_segments = ['event', strict_unicode(self.event_id), 'imported'] - filename = '{}-{}'.format(id_, filename) + path_segments = ['event', strict_str(self.event_id), 'imported'] + filename = f'{id_}-{filename}' path = posixpath.join(*(path_segments + [filename])) return path @@ -607,7 +605,7 @@ def _deserialize_object(self, table, data): # the exported data may contain only one event assert self.event_id is None insert_values['category_id'] = self.category_id - for col, value in data.iteritems(): + for col, value in data.items(): if isinstance(value, tuple): if value[0] == 'idref_set': assert set_idref is None @@ -624,7 +622,7 @@ def _deserialize_object(self, table, data): def _resolve_id_ref(value): return self._convert_value(colspec, value) rv = _exec_custom(import_custom[col], VALUE=value, RESOLVE_ID_REF=_resolve_id_ref) - assert rv.keys() == [col] + assert list(rv.keys()) == [col] insert_values[col] = rv[col] continue try: @@ -633,7 +631,7 @@ def _resolve_id_ref(value): deferred_idrefs[col] = exc.uuid except MissingUser as exc: if exc.skip: - click.secho('! Skipping row in {} due to missing user ({})'.format(table.fullname, exc.username), + click.secho(f'! Skipping row in {table.fullname} due to missing user ({exc.username})', fg='yellow') missing_user_skip = True else: @@ -662,7 +660,7 @@ def _resolve_id_ref(value): insert_values[pk_name] = pk_value = db.session.query(stmt).scalar() insert_values.update(self._process_file(pk_value, file_data)) else: - insert_values.update(self._process_file(unicode(uuid4()), file_data)) + insert_values.update(self._process_file(str(uuid4()), file_data)) if self.verbose and table.fullname in self.spec['verbose']: fmt = self.spec['verbose'][table.fullname] click.echo(fmt.format(**insert_values)) @@ -673,7 +671,7 @@ def _resolve_id_ref(value): self._set_idref(set_idref, _get_inserted_pk(res)) if is_event: self.event_id = _get_inserted_pk(res) - for col, uuid in deferred_idrefs.iteritems(): + for col, uuid in deferred_idrefs.items(): # store all the data needed to resolve a deferred ID reference # later once the ID is available self.deferred_idrefs[uuid].add((table, col, _get_inserted_pk(res))) diff --git a/indico/modules/events/export_test.py b/indico/modules/events/export_test.py index 5e41c795b85..bdcf237df6b 100644 --- a/indico/modules/events/export_test.py +++ b/indico/modules/events/export_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import tarfile import uuid @@ -24,7 +22,7 @@ from indico.util.date_time import as_utc -class _MockUUID(object): +class _MockUUID: def __init__(self): self.counter = 0 @@ -42,7 +40,7 @@ def reproducible_uuids(monkeypatch): @pytest.fixture def static_indico_version(monkeypatch): - monkeypatch.setattr('indico.__version__', b'1.3.3.7') + monkeypatch.setattr('indico.__version__', '1.3.3.7') @pytest.mark.usefixtures('reproducible_uuids', 'static_indico_version') @@ -61,13 +59,13 @@ def test_event_export(db, dummy_event, monkeypatch): export_event(dummy_event, f) f.seek(0) - with open(os.path.join(os.path.dirname(__file__), 'export_test_1.yaml'), 'r') as ref_file: + with open(os.path.join(os.path.dirname(__file__), 'export_test_1.yaml')) as ref_file: data_yaml_content = ref_file.read() # check composition of tarfile and data.yaml content with tarfile.open(fileobj=f) as tarf: assert tarf.getnames() == ['data.yaml'] - assert tarf.extractfile('data.yaml').read() == data_yaml_content + assert tarf.extractfile('data.yaml').read().decode() == data_yaml_content @pytest.mark.usefixtures('reproducible_uuids') @@ -94,9 +92,9 @@ def test_event_attachment_export(db, dummy_event, dummy_attachment): event_uid = objs[0][1]['id'][1] # check that the exported metadata contains all the right objects - assert [obj[0] for obj in objs] == [u'events.events', u'events.sessions', u'events.contributions', - u'events.contributions', u'attachments.folders', u'attachments.attachments', - u'attachments.files'] + assert [obj[0] for obj in objs] == ['events.events', 'events.sessions', 'events.contributions', + 'events.contributions', 'attachments.folders', 'attachments.attachments', + 'attachments.files'] # check that the attached file's metadata is included assert objs[5][1]['title'] == 'dummy_attachment' assert objs[5][1]['folder_id'] is not None @@ -109,15 +107,15 @@ def test_event_attachment_export(db, dummy_event, dummy_attachment): assert file_['md5'] == '5eb63bbbe01eeed093cb22bb8f5acdc3' # check that the file itself was included (and verify content) assert tarf.getnames() == ['00000000-0000-4000-8000-000000000013', 'data.yaml'] - assert tarf.extractfile('00000000-0000-4000-8000-000000000013').read() == 'hello world' + assert tarf.extractfile('00000000-0000-4000-8000-000000000013').read() == b'hello world' @pytest.mark.usefixtures('static_indico_version') def test_event_import(db, dummy_user): - with open(os.path.join(os.path.dirname(__file__), 'export_test_2.yaml'), 'r') as ref_file: + with open(os.path.join(os.path.dirname(__file__), 'export_test_2.yaml')) as ref_file: data_yaml_content = ref_file.read() - data_yaml = BytesIO(data_yaml_content.encode('utf-8')) + data_yaml = BytesIO(data_yaml_content.encode()) tar_buffer = BytesIO() # User should be matched by e-mail @@ -148,4 +146,4 @@ def test_event_import(db, dummy_user): attachment = folder.attachments[0] assert attachment.title == 'dummy_attachment' # Check that the actual file is accessible - assert attachment.file.open().read() == 'hello world' + assert attachment.file.open().read() == b'hello world' diff --git a/indico/modules/events/export_test_1.yaml b/indico/modules/events/export_test_1.yaml index d1759ee0f76..10da0dfcb48 100644 --- a/indico/modules/events/export_test_1.yaml +++ b/indico/modules/events/export_test_1.yaml @@ -1,152 +1,152 @@ -!!python/unicode 'indico_version': 1.3.3.7 -!!python/unicode 'objects': +indico_version: 1.3.3.7 +objects: - !!python/tuple - - !!python/unicode 'events.events' - - !!python/unicode 'access_key': !!python/unicode '' - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'created_dt': !!python/tuple + - events.events + - access_key: '' + address: '' + created_dt: !!python/tuple - datetime - '2017-08-24T00:00:00+00:00' - !!python/unicode 'creator_id': !!python/tuple - - !!python/unicode 'userref' - - !!python/unicode '00000000-0000-4000-8000-000000000001' - !!python/unicode 'custom_boa_id': null - !!python/unicode 'default_page_id': null - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'end_dt': !!python/tuple + creator_id: !!python/tuple + - userref + - 00000000-0000-4000-8000-000000000001 + custom_boa_id: null + default_page_id: null + description: '' + end_dt: !!python/tuple - datetime - '2017-08-24T12:00:00+00:00' - !!python/unicode 'id': !!python/tuple - - !!python/unicode 'idref_set' - - !!python/unicode '00000000-0000-4000-8000-000000000000' - !!python/unicode 'is_deleted': false - !!python/unicode 'is_locked': false - !!python/unicode 'keywords': [] - !!python/unicode 'last_friendly_contribution_id': 2 - !!python/unicode 'last_friendly_registration_id': 0 - !!python/unicode 'last_friendly_session_id': 1 - !!python/unicode 'logo': null - !!python/unicode 'logo_metadata': null - !!python/unicode 'map_url': !!python/unicode '' - !!python/unicode 'no_access_contact': !!python/unicode '' - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000000 + is_deleted: false + is_locked: false + keywords: [] + last_friendly_contribution_id: 2 + last_friendly_registration_id: 0 + last_friendly_session_id: 1 + logo: null + logo_metadata: null + map_url: '' + no_access_contact: '' + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode - 1 - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'start_dt': !!python/tuple + room_name: '' + start_dt: !!python/tuple - datetime - '2017-08-24T10:00:00+00:00' - !!python/unicode 'stylesheet': null - !!python/unicode 'stylesheet_metadata': null - !!python/unicode 'timezone': !!python/unicode 'UTC' - !!python/unicode 'title': !!python/unicode 'dummy#0' - !!python/unicode 'type': !!python/object/apply:indico.modules.events.models.events.EventType + stylesheet: null + stylesheet_metadata: null + timezone: UTC + title: dummy#0 + type: !!python/object/apply:indico.modules.events.models.events.EventType - 2 - !!python/unicode 'venue_name': !!python/unicode '' - !!python/unicode 'visibility': null + venue_name: '' + visibility: null - !!python/tuple - - !!python/unicode 'events.sessions' - - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'background_color': !!python/unicode 'e3f2d3' - !!python/unicode 'code': !!python/unicode '' - !!python/unicode 'default_contribution_duration': !!python/object/apply:datetime.timedelta + - events.sessions + - address: '' + background_color: e3f2d3 + code: '' + default_contribution_duration: !!python/object/apply:datetime.timedelta - 0 - 1200 - 0 - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'event_id': !!python/tuple - - !!python/unicode 'idref' - - !!python/unicode '00000000-0000-4000-8000-000000000000' - !!python/unicode 'friendly_id': 1 - !!python/unicode 'id': !!python/tuple - - !!python/unicode 'idref_set' - - !!python/unicode '00000000-0000-4000-8000-000000000003' - !!python/unicode 'inherit_location': true - !!python/unicode 'is_deleted': true - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + description: '' + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000000 + friendly_id: 1 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000002 + inherit_location: true + is_deleted: true + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode - 1 - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'text_color': !!python/unicode '202020' - !!python/unicode 'title': !!python/unicode 'sd' - !!python/unicode 'type_id': null - !!python/unicode 'venue_name': !!python/unicode '' + room_name: '' + text_color: '202020' + title: sd + type_id: null + venue_name: '' - !!python/tuple - - !!python/unicode 'events.contributions' - - !!python/unicode 'abstract_id': null - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'board_number': !!python/unicode '' - !!python/unicode 'code': !!python/unicode '' - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'duration': !!python/object/apply:datetime.timedelta + - events.contributions + - abstract_id: null + address: '' + board_number: '' + code: '' + description: '' + duration: !!python/object/apply:datetime.timedelta - 0 - 1800 - 0 - !!python/unicode 'event_id': !!python/tuple - - !!python/unicode 'idref' - - !!python/unicode '00000000-0000-4000-8000-000000000000' - !!python/unicode 'friendly_id': 1 - !!python/unicode 'id': !!python/tuple - - !!python/unicode 'idref_set' - - !!python/unicode '00000000-0000-4000-8000-000000000004' - !!python/unicode 'inherit_location': true - !!python/unicode 'is_deleted': false - !!python/unicode 'keywords': [] - !!python/unicode 'last_friendly_subcontribution_id': 0 - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000000 + friendly_id: 1 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000004 + inherit_location: true + is_deleted: false + keywords: [] + last_friendly_subcontribution_id: 0 + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode - 1 - !!python/unicode 'render_mode': !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode + render_mode: !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode - 2 - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'session_block_id': null - !!python/unicode 'session_id': null - !!python/unicode 'title': !!python/unicode 'c1' - !!python/unicode 'track_id': null - !!python/unicode 'type_id': null - !!python/unicode 'venue_name': !!python/unicode '' + room_name: '' + session_block_id: null + session_id: null + title: c1 + track_id: null + type_id: null + venue_name: '' - !!python/tuple - - !!python/unicode 'events.contributions' - - !!python/unicode 'abstract_id': null - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'board_number': !!python/unicode '' - !!python/unicode 'code': !!python/unicode '' - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'duration': !!python/object/apply:datetime.timedelta + - events.contributions + - abstract_id: null + address: '' + board_number: '' + code: '' + description: '' + duration: !!python/object/apply:datetime.timedelta - 0 - 1800 - 0 - !!python/unicode 'event_id': !!python/tuple - - !!python/unicode 'idref' - - !!python/unicode '00000000-0000-4000-8000-000000000000' - !!python/unicode 'friendly_id': 2 - !!python/unicode 'id': !!python/tuple - - !!python/unicode 'idref_set' - - !!python/unicode '00000000-0000-4000-8000-000000000006' - !!python/unicode 'inherit_location': true - !!python/unicode 'is_deleted': true - !!python/unicode 'keywords': [] - !!python/unicode 'last_friendly_subcontribution_id': 0 - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000000 + friendly_id: 2 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000006 + inherit_location: true + is_deleted: true + keywords: [] + last_friendly_subcontribution_id: 0 + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode - 1 - !!python/unicode 'render_mode': !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode + render_mode: !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode - 2 - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'session_block_id': null - !!python/unicode 'session_id': !!python/tuple - - !!python/unicode 'idref' - - !!python/unicode '00000000-0000-4000-8000-000000000003' - !!python/unicode 'title': !!python/unicode 'c2' - !!python/unicode 'track_id': null - !!python/unicode 'type_id': null - !!python/unicode 'venue_name': !!python/unicode '' -!!python/unicode 'timestamp': 2017-08-24 09:00:00+00:00 -!!python/unicode 'users': - !!python/unicode '00000000-0000-4000-8000-000000000001': - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'affiliation': null - !!python/unicode 'all_emails': - - !!python/unicode '1337@example.com' - !!python/unicode 'email': !!python/unicode '1337@example.com' - !!python/unicode 'first_name': !!python/unicode 'Guinea' - !!python/unicode 'last_name': !!python/unicode 'Pig' - !!python/unicode 'phone': !!python/unicode '' - !!python/unicode 'title': !!python/object/apply:indico.modules.users.models.users.UserTitle + room_name: '' + session_block_id: null + session_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000002 + title: c2 + track_id: null + type_id: null + venue_name: '' +timestamp: 2017-08-24 09:00:00+00:00 +users: + 00000000-0000-4000-8000-000000000001: + address: '' + affiliation: null + all_emails: + - 1337@example.com + email: 1337@example.com + first_name: Guinea + last_name: Pig + phone: '' + title: !!python/object/apply:indico.modules.users.models.users.UserTitle - 0 diff --git a/indico/modules/events/export_test_2.yaml b/indico/modules/events/export_test_2.yaml index 3e4931be0fd..345615332e9 100644 --- a/indico/modules/events/export_test_2.yaml +++ b/indico/modules/events/export_test_2.yaml @@ -1,162 +1,218 @@ -!!python/unicode 'indico_version': 1.3.3.7 -!!python/unicode 'objects': +indico_version: 1.3.3.7 +objects: - !!python/tuple - - !!python/unicode 'events.events' - - !!python/unicode 'access_key': !!python/unicode '' - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'created_dt': !!python/tuple [datetime, '2017-08-24T15:28:42.652626+00:00'] - !!python/unicode 'creator_id': !!python/tuple [!!python/unicode 'userref', !!python/unicode '00000000-0000-4000-8000-00000000000a'] - !!python/unicode 'default_page_id': null - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'end_dt': !!python/tuple [datetime, '2017-08-24T12:00:00+00:00'] - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-000000000009'] - !!python/unicode 'is_deleted': false - !!python/unicode 'is_locked': false - !!python/unicode 'keywords': [] - !!python/unicode 'last_friendly_contribution_id': 2 - !!python/unicode 'last_friendly_registration_id': 0 - !!python/unicode 'last_friendly_session_id': 1 - !!python/unicode 'logo': null - !!python/unicode 'logo_metadata': null - !!python/unicode 'no_access_contact': !!python/unicode '' - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode [ - 1] - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'start_dt': !!python/tuple [datetime, '2017-08-24T10:00:00+00:00'] - !!python/unicode 'stylesheet': null - !!python/unicode 'stylesheet_metadata': null - !!python/unicode 'timezone': !!python/unicode 'UTC' - !!python/unicode 'title': !!python/unicode 'dummy#0' - !!python/unicode 'type': !!python/object/apply:indico.modules.events.models.events.EventType [ - 2] - !!python/unicode 'venue_name': !!python/unicode '' - !!python/unicode 'visibility': null + - events.events + - access_key: '' + address: '' + created_dt: !!python/tuple + - datetime + - '2017-08-24T15:28:42.652626+00:00' + creator_id: !!python/tuple + - userref + - 00000000-0000-4000-8000-00000000000a + default_page_id: null + description: '' + end_dt: !!python/tuple + - datetime + - '2017-08-24T12:00:00+00:00' + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000009 + is_deleted: false + is_locked: false + keywords: [] + last_friendly_contribution_id: 2 + last_friendly_registration_id: 0 + last_friendly_session_id: 1 + logo: null + logo_metadata: null + no_access_contact: '' + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + - 1 + room_name: '' + start_dt: !!python/tuple + - datetime + - '2017-08-24T10:00:00+00:00' + stylesheet: null + stylesheet_metadata: null + timezone: UTC + title: dummy#0 + type: !!python/object/apply:indico.modules.events.models.events.EventType + - 2 + venue_name: '' + visibility: null - !!python/tuple - - !!python/unicode 'events.sessions' - - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'background_color': !!python/unicode 'e3f2d3' - !!python/unicode 'code': !!python/unicode '' - !!python/unicode 'default_contribution_duration': !!python/object/apply:datetime.timedelta [ - 0, 1200, 0] - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'event_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000009'] - !!python/unicode 'friendly_id': 1 - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-00000000000c'] - !!python/unicode 'inherit_location': true - !!python/unicode 'is_deleted': true - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode [ - 1] - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'text_color': !!python/unicode '202020' - !!python/unicode 'title': !!python/unicode 'sd' - !!python/unicode 'type_id': null - !!python/unicode 'venue_name': !!python/unicode '' + - events.sessions + - address: '' + background_color: e3f2d3 + code: '' + default_contribution_duration: !!python/object/apply:datetime.timedelta + - 0 + - 1200 + - 0 + description: '' + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000009 + friendly_id: 1 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-00000000000c + inherit_location: true + is_deleted: true + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + - 1 + room_name: '' + text_color: '202020' + title: sd + type_id: null + venue_name: '' - !!python/tuple - - !!python/unicode 'events.contributions' - - !!python/unicode 'abstract_id': null - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'board_number': !!python/unicode '' - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'duration': !!python/object/apply:datetime.timedelta [0, 1800, - 0] - !!python/unicode 'event_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000009'] - !!python/unicode 'friendly_id': 2 - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-00000000000d'] - !!python/unicode 'inherit_location': true - !!python/unicode 'is_deleted': true - !!python/unicode 'keywords': [] - !!python/unicode 'last_friendly_subcontribution_id': 0 - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode [ - 1] - !!python/unicode 'render_mode': !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode [ - 2] - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'session_block_id': null - !!python/unicode 'session_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-00000000000c'] - !!python/unicode 'title': !!python/unicode 'c2' - !!python/unicode 'track_id': null - !!python/unicode 'type_id': null - !!python/unicode 'venue_name': !!python/unicode '' + - events.contributions + - abstract_id: null + address: '' + board_number: '' + description: '' + duration: !!python/object/apply:datetime.timedelta + - 0 + - 1800 + - 0 + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000009 + friendly_id: 2 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-00000000000d + inherit_location: true + is_deleted: true + keywords: [] + last_friendly_subcontribution_id: 0 + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + - 1 + render_mode: !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode + - 2 + room_name: '' + session_block_id: null + session_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-00000000000c + title: c2 + track_id: null + type_id: null + venue_name: '' - !!python/tuple - - !!python/unicode 'events.contributions' - - !!python/unicode 'abstract_id': null - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'board_number': !!python/unicode '' - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'duration': !!python/object/apply:datetime.timedelta [0, 1800, - 0] - !!python/unicode 'event_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000009'] - !!python/unicode 'friendly_id': 1 - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-000000000010'] - !!python/unicode 'inherit_location': true - !!python/unicode 'is_deleted': false - !!python/unicode 'keywords': [] - !!python/unicode 'last_friendly_subcontribution_id': 0 - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode [ - 1] - !!python/unicode 'render_mode': !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode [ - 2] - !!python/unicode 'room_name': !!python/unicode '' - !!python/unicode 'session_block_id': null - !!python/unicode 'session_id': null - !!python/unicode 'title': !!python/unicode 'c1' - !!python/unicode 'track_id': null - !!python/unicode 'type_id': null - !!python/unicode 'venue_name': !!python/unicode '' + - events.contributions + - abstract_id: null + address: '' + board_number: '' + description: '' + duration: !!python/object/apply:datetime.timedelta + - 0 + - 1800 + - 0 + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000009 + friendly_id: 1 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000010 + inherit_location: true + is_deleted: false + keywords: [] + last_friendly_subcontribution_id: 0 + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + - 1 + render_mode: !!python/object/apply:indico.core.db.sqlalchemy.descriptions.RenderMode + - 2 + room_name: '' + session_block_id: null + session_id: null + title: c1 + track_id: null + type_id: null + venue_name: '' - !!python/tuple - - !!python/unicode 'attachments.folders' - - !!python/unicode 'category_id': null - !!python/unicode 'contribution_id': null - !!python/unicode 'description': !!python/unicode 'a dummy folder' - !!python/unicode 'event_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000009'] - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-000000000014'] - !!python/unicode 'is_always_visible': true - !!python/unicode 'is_default': false - !!python/unicode 'is_deleted': false - !!python/unicode 'link_type': !!python/object/apply:indico.core.db.sqlalchemy.links.LinkType [ - 2] - !!python/unicode 'linked_event_id': !!python/tuple [!!python/unicode 'idref', - !!python/unicode '00000000-0000-4000-8000-000000000009'] - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode [ - 1] - !!python/unicode 'session_id': null - !!python/unicode 'subcontribution_id': null - !!python/unicode 'title': !!python/unicode 'dummy_folder' + - attachments.folders + - category_id: null + contribution_id: null + description: a dummy folder + event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000009 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000014 + is_always_visible: true + is_default: false + is_deleted: false + link_type: !!python/object/apply:indico.core.db.sqlalchemy.links.LinkType + - 2 + linked_event_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000009 + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + - 1 + session_id: null + subcontribution_id: null + title: dummy_folder - !!python/tuple - - !!python/unicode 'attachments.attachments' - - !!python/unicode 'description': !!python/unicode '' - !!python/unicode 'file_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000018'] - !!python/unicode 'folder_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000014'] - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-000000000017'] - !!python/unicode 'is_deleted': false - !!python/unicode 'link_url': null - !!python/unicode 'modified_dt': !!python/tuple [datetime, '2017-08-24T15:28:42.693225+00:00'] - !!python/unicode 'protection_mode': !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode [ - 1] - !!python/unicode 'title': !!python/unicode 'dummy_attachment' - !!python/unicode 'type': !!python/object/apply:indico.modules.attachments.models.attachments.AttachmentType [ - 1] - !!python/unicode 'user_id': !!python/tuple [!!python/unicode 'userref', !!python/unicode '00000000-0000-4000-8000-00000000000a'] + - attachments.attachments + - description: '' + file_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000018 + folder_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000014 + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000017 + is_deleted: false + link_url: null + modified_dt: !!python/tuple + - datetime + - '2017-08-24T15:28:42.693225+00:00' + protection_mode: !!python/object/apply:indico.core.db.sqlalchemy.protection.ProtectionMode + - 1 + title: dummy_attachment + type: !!python/object/apply:indico.modules.attachments.models.attachments.AttachmentType + - 1 + user_id: !!python/tuple + - userref + - 00000000-0000-4000-8000-00000000000a - !!python/tuple - - !!python/unicode 'attachments.files' - - !!python/unicode '__file__': !!python/tuple - - !!python/unicode 'file' - - {!!python/unicode 'content_type': !!python/unicode 'text/plain', !!python/unicode 'filename': !!python/unicode 'dummy_file.txt', - !!python/unicode 'md5': !!python/unicode '5eb63bbbe01eeed093cb22bb8f5acdc3', !!python/unicode 'size': !!python/long '11', - !!python/unicode 'uuid': !!python/unicode '00000000-0000-4000-8000-00000000001c'} - !!python/unicode 'attachment_id': !!python/tuple [!!python/unicode 'idref', !!python/unicode '00000000-0000-4000-8000-000000000017'] - !!python/unicode 'created_dt': !!python/tuple [datetime, '2017-08-24T15:28:42.691791+00:00'] - !!python/unicode 'id': !!python/tuple [!!python/unicode 'idref_set', !!python/unicode '00000000-0000-4000-8000-000000000018'] - !!python/unicode 'user_id': !!python/tuple [!!python/unicode 'userref', !!python/unicode '00000000-0000-4000-8000-00000000000a'] -!!python/unicode 'timestamp': 2017-08-24 15:28:43.011442+00:00 -!!python/unicode 'users': - !!python/unicode '00000000-0000-4000-8000-00000000000a': - !!python/unicode 'address': !!python/unicode '' - !!python/unicode 'affiliation': null - !!python/unicode 'all_emails': [!!python/unicode '1337@example.com'] - !!python/unicode 'email': !!python/unicode '1337@example.com' - !!python/unicode 'first_name': !!python/unicode 'Guinea' - !!python/unicode 'last_name': !!python/unicode 'Pig' - !!python/unicode 'phone': !!python/unicode '' - !!python/unicode 'title': !!python/object/apply:indico.modules.users.models.users.UserTitle [ - 0] + - attachments.files + - __file__: !!python/tuple + - file + - content_type: text/plain + filename: dummy_file.txt + md5: 5eb63bbbe01eeed093cb22bb8f5acdc3 + size: 11 + uuid: 00000000-0000-4000-8000-00000000001c + attachment_id: !!python/tuple + - idref + - 00000000-0000-4000-8000-000000000017 + created_dt: !!python/tuple + - datetime + - '2017-08-24T15:28:42.691791+00:00' + id: !!python/tuple + - idref_set + - 00000000-0000-4000-8000-000000000018 + user_id: !!python/tuple + - userref + - 00000000-0000-4000-8000-00000000000a +timestamp: 2017-08-24 15:28:43.011442+00:00 +users: + 00000000-0000-4000-8000-00000000000a: + address: '' + affiliation: null + all_emails: + - 1337@example.com + email: 1337@example.com + first_name: Guinea + last_name: Pig + phone: '' + title: !!python/object/apply:indico.modules.users.models.users.UserTitle + - 0 diff --git a/indico/modules/events/features/__init__.py b/indico/modules/events/features/__init__.py index 68c481ca4f3..b61c18bef81 100644 --- a/indico/modules/events/features/__init__.py +++ b/indico/modules/events/features/__init__.py @@ -1,13 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - -from flask import flash, request, session +from flask import flash, session from indico.core import signals from indico.core.logger import Logger @@ -39,7 +37,7 @@ def _check_feature_definitions(app, **kwargs): @signals.event.created.connect def _event_created(event, cloning, **kwargs): - from indico.modules.events.features.util import get_feature_definitions, get_enabled_features + from indico.modules.events.features.util import get_enabled_features, get_feature_definitions feature_definitions = get_feature_definitions() for feature in get_enabled_features(event): feature_definitions[feature].enabled(event, cloning) @@ -47,15 +45,14 @@ def _event_created(event, cloning, **kwargs): @signals.event.type_changed.connect def _event_type_changed(event, **kwargs): - from indico.modules.events.features.util import (get_enabled_features, get_disallowed_features, set_feature_enabled, - format_feature_names) + from indico.modules.events.features.util import (format_feature_names, get_disallowed_features, + get_enabled_features, set_feature_enabled) conflicting = get_enabled_features(event, only_explicit=True) & get_disallowed_features(event) if conflicting: for feature in conflicting: set_feature_enabled(event, feature, False) - if request.endpoint != 'api.jsonrpc': - # XXX: we cannot flash a message in the legacy js ajax editor for the event type. - # remove this check once we don't use it anymore (on the general settings page) - flash(ngettext('Feature disabled: {features} (not available for this event type)', - 'Features disabled: {features} (not available for this event type)', len(conflicting)) - .format(features=format_feature_names(conflicting)), 'warning') + # XXX: we cannot flash a message in the legacy js ajax editor for the event type. + # remove this check once we don't use it anymore (on the general settings page) + flash(ngettext('Feature disabled: {features} (not available for this event type)', + 'Features disabled: {features} (not available for this event type)', len(conflicting)) + .format(features=format_feature_names(conflicting)), 'warning') diff --git a/indico/modules/events/features/base.py b/indico/modules/events/features/base.py index 3c32e3be286..4b19040015a 100644 --- a/indico/modules/events/features/base.py +++ b/indico/modules/events/features/base.py @@ -1,17 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.features.util import get_feature_definitions from indico.util.decorators import cached_classproperty -class EventFeature(object): +class EventFeature: """Base class for event features. To create a new feature, subclass this class and register @@ -53,23 +51,21 @@ class EventFeature(object): @classmethod def is_default_for_event(cls, event): # pragma: no cover - """Checks if the feature should be enabled by default""" + """Check if the feature should be enabled by default.""" return False @classmethod def is_allowed_for_event(cls, event): # pragma: no cover - """Check if the feature can be enabled in an event""" + """Check if the feature can be enabled in an event.""" return True @classmethod def enabled(cls, event, cloning): # pragma: no cover - """Called when the feature is enabled for an event""" - pass + """Called when the feature is enabled for an event.""" @classmethod def disabled(cls, event): # pragma: no cover - """Called when the feature is disabled for an event""" - pass + """Called when the feature is disabled for an event.""" @cached_classproperty @classmethod @@ -96,4 +92,4 @@ def required_by_deep(cls): this feature. """ # This is not very efficient, but it runs exactly one on a not-very-large set - return {feature.name for feature in get_feature_definitions().itervalues() if cls.name in feature.requires_deep} + return {feature.name for feature in get_feature_definitions().values() if cls.name in feature.requires_deep} diff --git a/indico/modules/events/features/blueprint.py b/indico/modules/events/features/blueprint.py index d21ed45daf8..a278d097d9e 100644 --- a/indico/modules/events/features/blueprint.py +++ b/indico/modules/events/features/blueprint.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.features.controllers import RHFeatures, RHSwitchFeature from indico.web.flask.wrappers import IndicoBlueprint _bp = IndicoBlueprint('event_features', __name__, template_folder='templates', - virtual_template_folder='events/features', url_prefix='/event//manage/features') + virtual_template_folder='events/features', url_prefix='/event//manage/features') _bp.add_url_rule('/', 'index', RHFeatures) _bp.add_url_rule('/', 'switch', RHSwitchFeature, methods=('PUT', 'DELETE')) diff --git a/indico/modules/events/features/controllers.py b/indico/modules/events/features/controllers.py index 7827cf6e8f5..21e01038546 100644 --- a/indico/modules/events/features/controllers.py +++ b/indico/modules/events/features/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, request, session from werkzeug.exceptions import Forbidden from wtforms.fields import BooleanField @@ -29,12 +27,12 @@ class RHFeaturesBase(RHManageEventBase): class RHFeatures(RHFeaturesBase): - """Shows the list of available event features""" + """Show the list of available event features.""" def _make_form(self): - form_class = type(b'FeaturesForm', (IndicoForm,), {}) + form_class = type('FeaturesForm', (IndicoForm,), {}) disallowed = get_disallowed_features(self.event) - for name, feature in sorted(get_feature_definitions().iteritems(), key=lambda x: x[1].friendly_name): + for name, feature in sorted(get_feature_definitions().items(), key=lambda x: x[1].friendly_name): if name in disallowed: continue field = BooleanField(feature.friendly_name, widget=SwitchWidget(), description=feature.description) @@ -49,7 +47,7 @@ def _process(self): class RHSwitchFeature(RHFeaturesBase): - """Enables/disables a feature""" + """Enable/disable a feature.""" def render_event_menu(self): return render_sidemenu('event-management-sidemenu', active_item=WPFeatures.sidemenu_option, @@ -68,7 +66,7 @@ def _process_PUT(self): .format(features=format_feature_names(changed)), 'success') logger.info("Feature '%s' for event %s enabled by %s", feature.name, self.event, session.user) self.event.log(EventLogRealm.management, EventLogKind.positive, 'Features', - 'Enabled {}'.format(feature.friendly_name), session.user) + f'Enabled {feature.friendly_name}', session.user) return jsonify_data(enabled=True, event_menu=self.render_event_menu(), changed=list(changed)) def _process_DELETE(self): @@ -82,5 +80,5 @@ def _process_DELETE(self): .format(features=format_feature_names(changed)), 'warning') logger.info("Feature '%s' for event %s disabled by %s", feature.name, self.event, session.user) self.event.log(EventLogRealm.management, EventLogKind.negative, 'Features', - 'Disabled {}'.format(feature.friendly_name), session.user) + f'Disabled {feature.friendly_name}', session.user) return jsonify_data(enabled=False, event_menu=self.render_event_menu(), changed=list(changed)) diff --git a/indico/modules/events/features/templates/features.html b/indico/modules/events/features/templates/features.html index 458096fcdef..1764ba1311f 100644 --- a/indico/modules/events/features/templates/features.html +++ b/indico/modules/events/features/templates/features.html @@ -32,7 +32,7 @@ }, href: function() { return build_url(urlTemplate, { - confId: {{ event.id }}, + event_id: {{ event.id }}, feature: this.name }); } diff --git a/indico/modules/events/features/util.py b/indico/modules/events/features/util.py index 3413a61b739..7f179f694bc 100644 --- a/indico/modules/events/features/util.py +++ b/indico/modules/events/features/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from itertools import chain from werkzeug.exceptions import NotFound @@ -19,31 +17,31 @@ def get_feature_definitions(): - """Gets a dict containing all feature definitions""" + """Get a dict containing all feature definitions.""" return named_objects_from_signal(signals.event.get_feature_definitions.send(), plugin_attr='plugin') def get_feature_definition(name): - """Gets a feature definition""" + """Get a feature definition.""" try: return get_feature_definitions()[name] except KeyError: - raise RuntimeError('Feature does not exist: {}'.format(name)) + raise RuntimeError(f'Feature does not exist: {name}') def get_enabled_features(event, only_explicit=False): - """Returns a set of enabled feature names for an event""" + """Return a set of enabled feature names for an event.""" enabled_features = features_event_settings.get(event, 'enabled') if enabled_features is not None: return set(enabled_features) elif only_explicit: return set() else: - return {name for name, feature in get_feature_definitions().iteritems() if feature.is_default_for_event(event)} + return {name for name, feature in get_feature_definitions().items() if feature.is_default_for_event(event)} def set_feature_enabled(event, name, state): - """Enables/disables a feature for an event + """Enable/disable a feature for an event. :param event: The event. :param name: The name of the feature. @@ -81,14 +79,14 @@ def get_disallowed_features(event): for an event. """ disallowed = {feature - for feature in get_feature_definitions().itervalues() + for feature in get_feature_definitions().values() if not feature.is_allowed_for_event(event)} indirectly_disallowed = set(chain.from_iterable(feature.required_by_deep for feature in disallowed)) return indirectly_disallowed | {f.name for f in disallowed} def is_feature_enabled(event, name): - """Checks if a feature is enabled for an event. + """Check if a feature is enabled for an event. :param event: The event (or event ID) to check. :param name: The name of the feature. @@ -98,23 +96,23 @@ def is_feature_enabled(event, name): if enabled_features is not None: return feature.name in enabled_features else: - if isinstance(event, (basestring, int, long)): + if isinstance(event, (str, int)): event = Event.get(event) return event and feature.is_default_for_event(event) def require_feature(event, name): - """Raises a NotFound error if a feature is not enabled + """Raise a NotFound error if a feature is not enabled. :param event: The event (or event ID) to check. :param name: The name of the feature. """ if not is_feature_enabled(event, name): feature = get_feature_definition(name) - raise NotFound("The '{}' feature is not enabled for this event.".format(feature.friendly_name)) + raise NotFound(f"The '{feature.friendly_name}' feature is not enabled for this event.") def format_feature_names(names): - return ', '.join(sorted(unicode(f.friendly_name) - for f in get_feature_definitions().itervalues() + return ', '.join(sorted(str(f.friendly_name) + for f in get_feature_definitions().values() if f.name in names)) diff --git a/indico/modules/events/features/views.py b/indico/modules/events/features/views.py index 081cf1b1217..9cd7be54d78 100644 --- a/indico/modules/events/features/views.py +++ b/indico/modules/events/features/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement diff --git a/indico/modules/events/fields.py b/indico/modules/events/fields.py index 7344b87c514..b817e809c5b 100644 --- a/indico/modules/events/fields.py +++ b/indico/modules/events/fields.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import json from sqlalchemy import inspect @@ -35,11 +33,11 @@ def __init__(self, *args, **kwargs): self.reference_class = kwargs.pop('reference_class') self.fields = [{'id': 'type', 'caption': _("Type"), 'type': 'select', 'required': True}, {'id': 'value', 'caption': _("Value"), 'type': 'text', 'required': True}] - self.choices = {'type': {unicode(r.id): r.name for r in ReferenceType.find_all()}} - super(ReferencesField, self).__init__(*args, uuid_field='id', uuid_field_opaque=True, **kwargs) + self.choices = {'type': {str(r.id): r.name for r in ReferenceType.query}} + super().__init__(*args, uuid_field='id', uuid_field_opaque=True, **kwargs) def process_formdata(self, valuelist): - super(ReferencesField, self).process_formdata(valuelist) + super().process_formdata(valuelist) if valuelist: existing = {x.id: x for x in self.object_data or ()} data = [] @@ -59,22 +57,22 @@ def process_formdata(self, valuelist): self.data = data def pre_validate(self, form): - super(ReferencesField, self).pre_validate(form) + super().pre_validate(form) for reference in self.serialized_data: if reference['type'] not in self.choices['type']: - raise ValueError(u'Invalid type choice: {}'.format(reference['type'])) + raise ValueError('Invalid type choice: {}'.format(reference['type'])) def _value(self): if not self.data: return [] else: - return [{'id': r.id, 'type': unicode(r.reference_type_id), 'value': r.value} for r in self.data] + return [{'id': r.id, 'type': str(r.reference_type_id), 'value': r.value} for r in self.data] class EventPersonListField(PrincipalListField): - """A field that lets you select a list Indico user and EventPersons + """A field that lets you select a list Indico user and EventPersons. - Requires its form to have an event set. + This requires its form to have an event set. """ #: Whether new event persons created by the field should be @@ -83,7 +81,7 @@ class EventPersonListField(PrincipalListField): def __init__(self, *args, **kwargs): self.event_person_conversions = {} - super(EventPersonListField, self).__init__(*args, allow_groups=False, allow_external_users=True, **kwargs) + super().__init__(*args, allow_groups=False, allow_external_users=True, **kwargs) @property def event(self): @@ -101,7 +99,7 @@ def _serialize_principal(self, principal): if isinstance(principal, dict): return principal if not isinstance(principal, EventPerson): - return super(EventPersonListField, self)._serialize_principal(principal) + return super()._serialize_principal(principal) return serialize_event_person(principal) def process_formdata(self, valuelist): @@ -125,7 +123,7 @@ class PersonLinkListFieldBase(EventPersonListField): widget = None def __init__(self, *args, **kwargs): - super(PersonLinkListFieldBase, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.object = getattr(kwargs['_form'], self.linked_object_attr, None) @no_autoflush @@ -141,7 +139,7 @@ def _get_person_link(self, data, extra_data=None): person_data.update(extra_data) person_link = None if self.object and inspect(person).persistent: - person_link = self.person_link_cls.find_first(person=person, object=self.object) + person_link = self.person_link_cls.query.filter_by(person=person, object=self.object).first() if not person_link: person_link = self.person_link_cls(person=person) person_link.populate_from_dict(person_data) @@ -158,9 +156,9 @@ def _get_person_link(self, data, extra_data=None): def _serialize_principal(self, principal): if not isinstance(principal, PersonLinkBase): - return super(PersonLinkListFieldBase, self)._serialize_principal(principal) + return super()._serialize_principal(principal) if principal.id is None: - return super(PersonLinkListFieldBase, self)._serialize_principal(principal.person) + return super()._serialize_principal(principal.person) else: return self._serialize_person_link(principal) @@ -172,7 +170,7 @@ def _value(self): class EventPersonLinkListField(PersonLinkListFieldBase): - """A field to manage event's chairpersons""" + """A field to manage event's chairpersons.""" person_link_cls = EventPersonLink linked_object_attr = 'event' @@ -181,7 +179,7 @@ class EventPersonLinkListField(PersonLinkListFieldBase): def __init__(self, *args, **kwargs): self.allow_submitters = True self.default_is_submitter = kwargs.pop('default_is_submitter', True) - super(EventPersonLinkListField, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def _convert_data(self, data): return {self._get_person_link(x): x.pop('isSubmitter', self.default_is_submitter) for x in data} @@ -193,7 +191,7 @@ def _serialize_person_link(self, principal, extra_data=None): return data def pre_validate(self, form): - super(EventPersonLinkListField, self).pre_validate(form) + super().pre_validate(form) persons = set() for person_link in self.data: if person_link.person in persons: @@ -205,9 +203,9 @@ class IndicoThemeSelectField(SelectField): def __init__(self, *args, **kwargs): allow_default = kwargs.pop('allow_default', False) event_type = kwargs.pop('event_type').name - super(IndicoThemeSelectField, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.choices = sorted([(tid, theme['title']) - for tid, theme in theme_settings.get_themes_for(event_type).viewitems()], + for tid, theme in theme_settings.get_themes_for(event_type).items()], key=lambda x: x[1].lower()) if allow_default: self.choices.insert(0, ('', _('Category default'))) @@ -220,4 +218,4 @@ class RatingReviewField(RadioField): def __init__(self, *args, **kwargs): self.question = kwargs.pop('question') self.rating_range = kwargs.pop('rating_range') - super(RatingReviewField, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) diff --git a/indico/modules/events/forms.py b/indico/modules/events/forms.py index eebce350740..e86790f0b6a 100644 --- a/indico/modules/events/forms.py +++ b/indico/modules/events/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import time from flask import session @@ -40,10 +38,10 @@ class ReferenceTypeForm(IndicoForm): def __init__(self, *args, **kwargs): self.reference_type = kwargs.pop('reference_type', None) - super(ReferenceTypeForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def validate_name(self, field): - query = ReferenceType.find(db.func.lower(ReferenceType.name) == field.data.lower()) + query = ReferenceType.query.filter(db.func.lower(ReferenceType.name) == field.data.lower()) if self.reference_type: query = query.filter(ReferenceType.id != self.reference_type.id) if query.count(): @@ -56,11 +54,11 @@ def validate_url_template(self, field): class EventLabelForm(IndicoForm): title = StringField(_('Title'), [DataRequired()]) - color = SelectField(_('Color'), [DataRequired()], choices=zip(get_sui_colors(), get_sui_colors())) + color = SelectField(_('Color'), [DataRequired()], choices=list(zip(get_sui_colors(), get_sui_colors()))) def __init__(self, *args, **kwargs): self.event_label = kwargs.pop('event_label', None) - super(EventLabelForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def validate_title(self, field): query = EventLabel.query.filter(db.func.lower(EventLabel.title) == field.data.lower()) @@ -71,7 +69,7 @@ def validate_title(self, field): class EventCreationFormBase(IndicoForm): - category = CategoryField(_('Category'), [DataRequired()], allow_subcats=False, require_event_creation_rights=True) + category = CategoryField(_('Category'), [DataRequired()], require_event_creation_rights=True) title = StringField(_('Event title'), [DataRequired()]) timezone = IndicoTimezoneSelectField(_('Timezone'), [DataRequired()]) location_data = IndicoLocationField(_('Location'), allow_location_inheritance=False, edit_address=False) diff --git a/indico/modules/events/ical.py b/indico/modules/events/ical.py new file mode 100644 index 00000000000..b890a9e816f --- /dev/null +++ b/indico/modules/events/ical.py @@ -0,0 +1,130 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +import icalendar +from lxml import html +from lxml.etree import ParserError +from werkzeug.urls import url_parse + +from indico.core import signals +from indico.core.config import config +from indico.core.db.sqlalchemy.protection import ProtectionMode +from indico.util.date_time import now_utc +from indico.util.signals import values_from_signal + + +def generate_basic_component(entity, uid=None, url=None): + """Generate an iCalendar component with basic common properties. + + :param entity: Event/session/contribution where properties come from + :param uid: UID for the component + :param url: URL for the component (defaults to `entity.external_url`) + + :return: iCalendar event with basic properties + """ + component = icalendar.Event() + + component.add('dtstamp', now_utc(False)) + component.add('dtstart', entity.start_dt) + component.add('dtend', entity.end_dt) + component.add('summary', entity.title) + + if uid: + component.add('uid', uid) + + if url: + component.add('url', url) + elif hasattr(entity, 'external_url'): + component.add('url', entity.external_url) + + location = (f'{entity.room_name} ({entity.venue_name})' + if entity.venue_name and entity.room_name + else (entity.venue_name or entity.room_name)) + if location: + component.add('location', location) + + speaker_list = getattr(entity, 'person_links', []) + description = [] + if speaker_list: + speakers = [f'{x.full_name} ({x.affiliation})' if x.affiliation else x.full_name + for x in speaker_list] + description.append('Speakers: {}'.format(', '.join(speakers))) + if entity.description: + desc_text = str(entity.description) or '

' # get rid of RichMarkup + try: + description.append(str(html.fromstring(desc_text).text_content())) + except ParserError: + # this happens if desc_text only contains a html comment + pass + if description: + component.add('description', '\n'.join(description)) + + return component + + +def generate_event_component(event, user=None): + """Generate an event icalendar component from an Indico event.""" + uid = f'indico-event-{event.id}@{url_parse(config.BASE_URL).host}' + component = generate_basic_component(event, uid) + + # add contact information + contact_info = event.contact_emails + event.contact_phones + if contact_info: + component.add('contact', ';'.join(contact_info)) + + # add logo url if event is public + if event.effective_protection_mode == ProtectionMode.public and event.has_logo: + component.add('image', event.external_logo_url, {'VALUE': 'URI'}) + + # send description to plugins in case one wants to add anything to it + data = {'description': component.get('description', '')} + for update in values_from_signal( + signals.event.metadata_postprocess.send('ical-export', event=event, data=data, user=user), + as_list=True + ): + data.update(update) + component.add('description', data['description']) + + return component + + +def event_to_ical(event, user=None, detailed=False): + """Serialize an event into an ical. + + :param event: The event to serialize + :param user: The user who needs to be able to access the events + :param detailed: If True, iCal will include the event's contributions + """ + return events_to_ical([event], user, detailed) + + +def events_to_ical(events, user=None, detailed=False): + """Serialize multiple events into an ical. + + :param events: A list of events to serialize + :param user: The user who needs to be able to access the events + :param detailed: If True, iCal will include the event's contributions + """ + calendar = icalendar.Calendar() + calendar.add('version', '2.0') + calendar.add('prodid', '-//CERN//INDICO//EN') + + for event in events: + if not detailed: + component = generate_event_component(event, user) + calendar.add_component(component) + else: + from indico.modules.events.contributions.ical import generate_contribution_component + components = [ + generate_contribution_component(contrib) + for contrib in event.contributions + if contrib.start_dt + ] + for component in components: + calendar.add_component(component) + + return calendar.to_ical() diff --git a/indico/modules/events/layout/__init__.py b/indico/modules/events/layout/__init__.py index e53c2740cec..20cec0e84fd 100644 --- a/indico/modules/events/layout/__init__.py +++ b/indico/modules/events/layout/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from jinja2.filters import do_filesizeformat @@ -100,7 +98,7 @@ def _get_feature_definitions(sender, **kwargs): @signals.event_management.image_created.connect def _log_image_created(image, user, **kwargs): image.event.log(EventLogRealm.management, EventLogKind.positive, 'Layout', - 'Added image "{}"'.format(image.filename), user, data={ + f'Added image "{image.filename}"', user, data={ 'File name': image.filename, 'File type': image.content_type, 'File size': do_filesizeformat(image.size) @@ -110,7 +108,7 @@ def _log_image_created(image, user, **kwargs): @signals.event_management.image_deleted.connect def _log_image_deleted(image, user, **kwargs): image.event.log(EventLogRealm.management, EventLogKind.negative, 'Layout', - 'Deleted image "{}"'.format(image.filename), user, data={ + f'Deleted image "{image.filename}"', user, data={ 'File name': image.filename }) diff --git a/indico/modules/events/layout/blueprint.py b/indico/modules/events/layout/blueprint.py index aee2431a7b5..15dab857ee8 100644 --- a/indico/modules/events/layout/blueprint.py +++ b/indico/modules/events/layout/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.layout.compat import compat_image, compat_page from indico.modules.events.layout.controllers.images import RHImageDelete, RHImageDisplay, RHImages, RHImageUpload from indico.modules.events.layout.controllers.layout import (RHLayoutCSSDelete, RHLayoutCSSDisplay, RHLayoutCSSPreview, @@ -22,7 +20,7 @@ _bp = IndicoBlueprint('event_layout', __name__, template_folder='templates', - virtual_template_folder='events/layout', url_prefix='/event//manage/layout') + virtual_template_folder='events/layout', url_prefix='/event//manage/layout') _bp.add_url_rule('/', 'index', RHLayoutEdit, methods=('GET', 'POST')) _bp.add_url_rule('/timetable-theme-form', 'timetable_theme_form', RHLayoutTimetableThemeForm) @@ -45,21 +43,21 @@ _bp.add_url_rule('/images/', 'images', RHImages) _bp.add_url_rule('/images/upload', 'images_upload', RHImageUpload, methods=('POST',)) _bp.add_url_rule('/images/-', 'image_delete', RHImageDelete, methods=('DELETE',)) -_bp.add_url_rule('!/event//.css', 'css_display', RHLayoutCSSDisplay) +_bp.add_url_rule('!/event//.css', 'css_display', RHLayoutCSSDisplay) _bp_images = IndicoBlueprint('event_images', __name__, template_folder='templates', - virtual_template_folder='events/layout', url_prefix='/event/') + virtual_template_folder='events/layout', url_prefix='/event/') _bp_images.add_url_rule('/logo-.png', 'logo_display', RHLogoDisplay) _bp_images.add_url_rule('/images/-', 'image_display', RHImageDisplay) _bp_pages = IndicoBlueprint('event_pages', __name__, template_folder='templates', - virtual_template_folder='events/layout', url_prefix='/event/') + virtual_template_folder='events/layout', url_prefix='/event/') _bp_pages.add_url_rule('/page/', 'page_display', RHPageDisplay) _bp_pages.add_url_rule('/page/-', 'page_display', RHPageDisplay) -_compat_bp = IndicoBlueprint('compat_layout', __name__, url_prefix='/event/') +_compat_bp = IndicoBlueprint('compat_layout', __name__, url_prefix='/event/') _compat_bp.add_url_rule('!/internalPage.py', 'page_modpython', make_compat_redirect_func(_compat_bp, 'page', view_args_conv={'confId': 'event_id', 'pageId': 'legacy_page_id'})) diff --git a/indico/modules/events/layout/clone.py b/indico/modules/events/layout/clone.py index 4e1cfca2672..ad6be0b73c7 100644 --- a/indico/modules/events/layout/clone.py +++ b/indico/modules/events/layout/clone.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy.util.models import get_simple_column_attrs from indico.modules.events.cloning import EventCloner diff --git a/indico/modules/events/layout/compat.py b/indico/modules/events/layout/compat.py index aef8d3441e6..1019ec8319e 100644 --- a/indico/modules/events/layout/compat.py +++ b/indico/modules/events/layout/compat.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import current_app, redirect from indico.modules.events.layout.models.legacy_mapping import LegacyImageMapping, LegacyPageMapping @@ -16,12 +14,12 @@ @RHSimple.wrap_function def compat_page(**kwargs): - page = LegacyPageMapping.find(**kwargs).first_or_404().page + page = LegacyPageMapping.query.filter_by(**kwargs).first_or_404().page return redirect(url_for('event_pages.page_display', page), 302 if current_app.debug else 301) @RHSimple.wrap_function def compat_image(**kwargs): kwargs.pop('image_ext', None) - image = LegacyImageMapping.find(**kwargs).first_or_404().image + image = LegacyImageMapping.query.filter_by(**kwargs).first_or_404().image return redirect(url_for('event_images.image_display', image), 302 if current_app.debug else 301) diff --git a/indico/modules/events/layout/controllers/images.py b/indico/modules/events/layout/controllers/images.py index da7380b0c55..9dcbf698e61 100644 --- a/indico/modules/events/layout/controllers/images.py +++ b/indico/modules/events/layout/controllers/images.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import shutil from io import BytesIO @@ -53,7 +51,7 @@ def _process(self): data.seek(0) try: image_type = Image.open(data).format.lower() - except IOError: + except OSError: # Invalid image data continue data.seek(0) diff --git a/indico/modules/events/layout/controllers/layout.py b/indico/modules/events/layout/controllers/layout.py index 5c24fb99bcd..76d984919df 100644 --- a/indico/modules/events/layout/controllers/layout.py +++ b/indico/modules/events/layout/controllers/layout.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os from io import BytesIO @@ -30,7 +28,7 @@ from indico.modules.events.views import WPConferenceDisplay from indico.util.fs import secure_filename from indico.util.i18n import _ -from indico.util.string import crc32, to_unicode +from indico.util.string import crc32 from indico.web.flask.templating import get_template_module from indico.web.flask.util import send_file, url_for from indico.web.forms import fields as indico_fields @@ -47,19 +45,19 @@ def _make_theme_settings_form(event, theme): settings = theme_settings.themes[theme]['user_settings'] except KeyError: return None - form_class = type(b'ThemeSettingsForm', (IndicoForm,), {}) - for name, field_data in settings.iteritems(): + form_class = type('ThemeSettingsForm', (IndicoForm,), {}) + for name, field_data in settings.items(): field_type = field_data['type'] field_class = getattr(indico_fields, field_type, None) or getattr(wtforms_fields, field_type, None) if not field_class: - raise Exception('Invalid field type: {}'.format(field_type)) + raise Exception(f'Invalid field type: {field_type}') label = field_data['caption'] description = field_data.get('description') validators = [DataRequired()] if field_data.get('required') else [] field = field_class(label, validators, description=description, **field_data.get('kwargs', {})) setattr(form_class, name, field) - defaults = {name: field_data.get('defaults') for name, field_data in settings.iteritems()} + defaults = {name: field_data.get('defaults') for name, field_data in settings.items()} if theme == event.theme: defaults.update(layout_settings.get(event, 'timetable_theme_settings')) @@ -90,7 +88,7 @@ def _process_POST(self): ret = self._process_request() new_values = layout_settings.get_all(self.event) # Skip `timetable_theme_settings` as they are dynamically generated from themes.yaml - changes = {k: (old_values[k], v) for k, v in new_values.iteritems() + changes = {k: (old_values[k], v) for k, v in new_values.items() if old_values[k] != v and k != 'timetable_theme_settings'} if changes: form_cls = ConferenceLayoutForm if self.event.type_ == EventType.conference else LectureMeetingLayoutForm @@ -130,7 +128,7 @@ def _process_conference(self): layout_settings.set(self.event, 'timetable_theme_settings', tt_theme_settings_form.data) else: layout_settings.delete(self.event, 'timetable_theme_settings') - data = {unicode(key): value for key, value in form.data.iteritems() if key in layout_settings.defaults} + data = {str(key): value for key, value in form.data.items() if key in layout_settings.defaults} layout_settings.set_multi(self.event, data) if form.theme.data == '_custom': layout_settings.set(self.event, 'use_custom_css', True) @@ -151,7 +149,7 @@ def _process(self): f = request.files['logo'] try: img = Image.open(f) - except IOError: + except OSError: flash(_('You cannot upload this file as a logo.'), 'error') return jsonify_data(content=None) if img.format.lower() not in {'jpeg', 'png', 'gif'}: @@ -189,7 +187,7 @@ def _process(self): class RHLayoutCSSUpload(RHLayoutBase): def _process(self): f = request.files['css_file'] - self.event.stylesheet = to_unicode(f.read()).strip() + self.event.stylesheet = str(f.read()).strip() self.event.stylesheet_metadata = { 'hash': crc32(self.event.stylesheet), 'size': len(self.event.stylesheet), @@ -259,5 +257,5 @@ def _check_access(self): def _process(self): if not self.event.has_stylesheet: raise NotFound - data = BytesIO(self.event.stylesheet.encode('utf-8')) + data = BytesIO(self.event.stylesheet.encode()) return send_file(self.event.stylesheet_metadata['filename'], data, mimetype='text/css', conditional=True) diff --git a/indico/modules/events/layout/controllers/menu.py b/indico/modules/events/layout/controllers/menu.py index 1bed13f5c8d..c7f57e0032a 100644 --- a/indico/modules/events/layout/controllers/menu.py +++ b/indico/modules/events/layout/controllers/menu.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from itertools import count from flask import flash, jsonify, request, session @@ -130,7 +128,7 @@ def _process(self): raise BadRequest('Menu entry "{0}" cannot be moved to another menu: Invalid type "{0.type.name}".' .format(self.entry)) if self.entry.is_root and self.entry.children: - raise BadRequest('Menu entry "{0}" cannot be moved to another menu: Entry has nested entries.' + raise BadRequest('Menu entry "{}" cannot be moved to another menu: Entry has nested entries.' .format(self.entry)) parent_entry = None @@ -141,7 +139,7 @@ def _process(self): MenuEntry.parent_id.is_(None)) .first()) if not parent_entry: - raise BadRequest('New parent entry not found for Menu entry "{0}".'.format(self.entry)) + raise BadRequest(f'New parent entry not found for Menu entry "{self.entry}".') self.entry.insert(parent_entry, position) @@ -211,7 +209,7 @@ def _process(self): class RHMenuDeleteEntry(RHMenuEntryEditBase): def _process(self): if self.entry.type not in (MenuEntryType.user_link, MenuEntryType.page, MenuEntryType.separator): - raise BadRequest('Menu entry of type {} cannot be deleted'.format(self.entry.type.name)) + raise BadRequest(f'Menu entry of type {self.entry.type.name} cannot be deleted') position_gen = count(self.entry.position) if self.entry.children: diff --git a/indico/modules/events/layout/forms.py b/indico/modules/events/layout/forms.py index 02eef4502eb..899bb638792 100644 --- a/indico/modules/events/layout/forms.py +++ b/indico/modules/events/layout/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms.fields import BooleanField, SelectField, TextAreaField from wtforms.fields.html5 import URLField from wtforms.fields.simple import StringField @@ -30,12 +28,12 @@ def _get_timetable_theme_choices(event): - it = ((tid, data['title']) for tid, data in theme_settings.get_themes_for(event.type).viewitems()) + it = ((tid, data['title']) for tid, data in theme_settings.get_themes_for(event.type).items()) return sorted(it, key=lambda x: x[1].lower()) def _get_conference_theme_choices(): - plugin_themes = [(k, v[1]) for k, v in get_plugin_conference_themes().iteritems()] + plugin_themes = [(k, v[1]) for k, v in get_plugin_conference_themes().items()] return THEMES + sorted(plugin_themes, key=lambda x: x[1].lower()) @@ -60,7 +58,7 @@ def build_field_metadata(cls, field): @property def log_fields_metadata(self): - return {k: self.build_field_metadata(v) for k, v in self._fields.iteritems()} + return {k: self.build_field_metadata(v) for k, v in self._fields.items()} class ConferenceLayoutForm(LoggedLayoutForm): @@ -102,7 +100,7 @@ class ConferenceLayoutForm(LoggedLayoutForm): def __init__(self, *args, **kwargs): self.event = kwargs.pop('event') - super(ConferenceLayoutForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.timetable_theme.choices = [('', _('Default'))] + _get_timetable_theme_choices(self.event) self.theme.choices = _get_conference_theme_choices() @@ -118,7 +116,7 @@ class LectureMeetingLayoutForm(LoggedLayoutForm): def __init__(self, *args, **kwargs): event = kwargs.pop('event') - super(LectureMeetingLayoutForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.timetable_theme.choices = _get_timetable_theme_choices(event) @@ -133,7 +131,7 @@ class CSSForm(IndicoForm): get_metadata=get_css_file_data, handle_flashes=True) def __init__(self, *args, **kwargs): - super(CSSForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.css_file.description = _("If you want to fully customize your conference page you can create your own " "stylesheet and upload it. An example stylesheet can be downloaded " "here." @@ -147,7 +145,7 @@ class MenuBuiltinEntryForm(IndicoForm): def __init__(self, *args, **kwargs): entry = kwargs.pop('entry') - super(MenuBuiltinEntryForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.custom_title.description = _("If you customize the title, that title is used regardless of the user's " "language preference. The default title {title} is " "displayed in the user's language.").format(title=entry.default_data.title) @@ -182,7 +180,7 @@ class CSSSelectionForm(IndicoForm): def __init__(self, *args, **kwargs): event = kwargs.pop('event') - super(CSSSelectionForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.theme.choices = _get_conference_theme_choices() if event.has_stylesheet: custom = [('_custom', _("Custom CSS file ({name})").format(name=event.stylesheet_metadata['filename']))] diff --git a/indico/modules/events/layout/models/images.py b/indico/modules/events/layout/models/images.py index f52b7dea382..32f5c13ee0c 100644 --- a/indico/modules/events/layout/models/images.py +++ b/indico/modules/events/layout/models/images.py @@ -1,19 +1,17 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import posixpath from indico.core.config import config from indico.core.db import db from indico.core.storage import StoredFileMixin from indico.util.fs import secure_filename -from indico.util.string import return_ascii, strict_unicode +from indico.util.string import strict_str class ImageFile(StoredFileMixin, db.Model): @@ -54,13 +52,12 @@ def locator(self): return dict(self.event.locator, image_id=self.id, filename=self.filename) def _build_storage_path(self): - path_segments = ['event', strict_unicode(self.event.id), 'images'] + path_segments = ['event', strict_str(self.event.id), 'images'] self.assign_id() filename = '{}-{}'.format(self.id, secure_filename(self.filename, 'file')) path = posixpath.join(*(path_segments + [filename])) return config.ATTACHMENT_STORAGE, path - @return_ascii def __repr__(self): return ''.format( self.id, diff --git a/indico/modules/events/layout/models/legacy_mapping.py b/indico/modules/events/layout/models/legacy_mapping.py index 0abdca9797d..4d656bb9d3a 100644 --- a/indico/modules/events/layout/models/legacy_mapping.py +++ b/indico/modules/events/layout/models/legacy_mapping.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class LegacyImageMapping(db.Model): - """Legacy image id mapping + """Legacy image id mapping. Legacy images had event-unique numeric ids. Using this mapping we can resolve old ones to their new id. @@ -52,13 +50,12 @@ class LegacyImageMapping(db.Model): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'legacy_image_id', 'image_id') class LegacyPageMapping(db.Model): - """Legacy page id mapping + """Legacy page id mapping. Legacy pages had event-unique numeric ids. Using this mapping we can resolve old ones to their new id. @@ -98,6 +95,5 @@ class LegacyPageMapping(db.Model): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'legacy_page_id', 'page_id') diff --git a/indico/modules/events/layout/models/menu.py b/indico/modules/events/layout/models/menu.py index ffbc2a9c1d1..c1b8003d1a0 100644 --- a/indico/modules/events/layout/models/menu.py +++ b/indico/modules/events/layout/models/menu.py @@ -1,21 +1,19 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import g, session from sqlalchemy.orm import joinedload from werkzeug.utils import cached_property from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum +from indico.util.enum import RichIntEnum from indico.util.i18n import _ -from indico.util.string import format_repr, return_ascii, slugify, text_to_repr -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_repr, slugify, text_to_repr from indico.web.flask.util import url_for @@ -23,7 +21,9 @@ def _get_next_position(context): """Get the next menu entry position for the event.""" event_id = context.current_parameters['event_id'] parent_id = context.current_parameters['parent_id'] - res = db.session.query(db.func.max(MenuEntry.position)).filter_by(event_id=event_id, parent_id=parent_id).one() + res = (db.session.query(db.func.max(MenuEntry.position)) + .filter(MenuEntry.event_id == event_id, MenuEntry.parent_id == parent_id) + .one()) return (res[0] or 0) + 1 @@ -36,10 +36,10 @@ class MenuEntryType(RichIntEnum): page = 5 -class MenuEntryMixin(object): +class MenuEntryMixin: def __init__(self, **kwargs): event = kwargs.pop('event', kwargs.get('event')) - super(MenuEntryMixin, self).__init__(**kwargs) + super().__init__(**kwargs) # XXX: not calling this `event` since this one should NOT use # the relationship to avoid mixing data from different DB sessions # when updating/populating the menu (which happens in a separate @@ -66,11 +66,11 @@ def url(self): return None elif self.is_internal_link: data = self.default_data - if data.static_site and isinstance(data.static_site, basestring) and g.get('static_site'): + if data.static_site and isinstance(data.static_site, str) and g.get('static_site'): return data.static_site kwargs = dict(data.url_kwargs) if self.name == 'timetable': - from indico.modules.events. layout import layout_settings + from indico.modules.events.layout import layout_settings if layout_settings.get(self.event_ref, 'timetable_by_room'): kwargs['layout'] = 'room' if layout_settings.get(self.event_ref, 'timetable_detailed'): @@ -160,7 +160,6 @@ def localized_title(self): def locator(self): return dict(self.event_ref.locator, menu_entry_id=self.id) - @return_ascii def __repr__(self): return '<{}({}, {}, {}, position={})>'.format( type(self).__name__, @@ -173,7 +172,7 @@ def __repr__(self): class TransientMenuEntry(MenuEntryMixin): def __init__(self, event, is_enabled, name, position, children): - super(TransientMenuEntry, self).__init__(event=event) + super().__init__(event=event) self.is_enabled = is_enabled self.title = None self.name = name @@ -437,6 +436,5 @@ def locator(self): def is_default(self): return self.menu_entry.event.default_page_id == self.id - @return_ascii def __repr__(self): return format_repr(self, 'id', _text=text_to_repr(self.html, html=True)) diff --git a/indico/modules/events/layout/templates/image_list.html b/indico/modules/events/layout/templates/image_list.html index 2cf776147f0..67df2684bab 100644 --- a/indico/modules/events/layout/templates/image_list.html +++ b/indico/modules/events/layout/templates/image_list.html @@ -14,7 +14,7 @@ title="{% trans %}Delete image{% endtrans %}" data-href="{{ url_for('event_layout.image_delete', image) }}" data-method="DELETE" - data-confirm="{% trans filename=image.filename %}Are you sure you want to delete '{{filename}}'? {% endtrans %}"> + data-confirm="{% trans filename=image.filename %}Are you sure you want to delete '{{ filename }}'? {% endtrans %}"> diff --git a/indico/modules/events/layout/templates/layout_conference.html b/indico/modules/events/layout/templates/layout_conference.html index bd6c9c7251a..469efdc20c3 100644 --- a/indico/modules/events/layout/templates/layout_conference.html +++ b/indico/modules/events/layout/templates/layout_conference.html @@ -69,6 +69,8 @@

{% trans %}Event Logo{% endtrans %}

{% endcall %} + {% block after_forms %}{% endblock %} + {% endblock %} diff --git a/indico/modules/events/persons/util.py b/indico/modules/events/persons/util.py index 9ed44aecf19..2a364e104fc 100644 --- a/indico/modules/events/persons/util.py +++ b/indico/modules/events/persons/util.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.models.persons import EventPerson from indico.modules.users import User from indico.modules.users.models.users import UserTitle -from indico.util.user import principal_from_fossil +from indico.util.user import principal_from_identifier def create_event_person(event, create_untrusted_persons=False, **data): @@ -27,8 +25,7 @@ def get_event_person_for_user(event, user, create_untrusted_persons=False): return EventPerson.for_user(user, event, is_untrusted=create_untrusted_persons) -def get_event_person(event, data, create_untrusted_persons=False, allow_external=False, allow_emails=False, - allow_networks=False): +def get_event_person(event, data, create_untrusted_persons=False, allow_external=False): """Get an EventPerson from dictionary data. If there is already an event person in the same event and for the same user, @@ -48,13 +45,11 @@ def get_event_person(event, data, create_untrusted_persons=False, allow_external # We have no way to identify an existing event person with the provided information return create_event_person(event, create_untrusted_persons=create_untrusted_persons, **data) elif person_type == 'Avatar': - # XXX: existing_data - principal = principal_from_fossil(data, allow_pending=allow_external, allow_emails=allow_emails, - allow_networks=allow_networks) + principal = principal_from_identifier(data['identifier'], allow_external_users=allow_external) return get_event_person_for_user(event, principal, create_untrusted_persons=create_untrusted_persons) elif person_type == 'EventPerson': return event.persons.filter_by(id=data['id']).one() elif person_type == 'PersonLink': return event.persons.filter_by(id=data['personId']).one() else: - raise ValueError("Unknown person type '{}'".format(person_type)) + raise ValueError(f"Unknown person type '{person_type}'") diff --git a/indico/modules/events/persons/util_test.py b/indico/modules/events/persons/util_test.py index 79c0527b8c8..a6955237a97 100644 --- a/indico/modules/events/persons/util_test.py +++ b/indico/modules/events/persons/util_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import pytest from indico.modules.events.persons.util import create_event_person, get_event_person, get_event_person_for_user @@ -106,7 +104,7 @@ def test_get_event_person_edit(db, dummy_event, dummy_user): 'familyName': 'Doe', 'affiliation': 'ACME Inc.' } - person_1 = get_event_person(dummy_event, dict(data, _type='Avatar', id=dummy_user.id)) + person_1 = get_event_person(dummy_event, dict(data, _type='Avatar', identifier=f'User:{dummy_user.id}')) assert person_1.id is None assert person_1.user == dummy_user db.session.add(person_1) diff --git a/indico/modules/events/persons/views.py b/indico/modules/events/persons/views.py index 1c5062185ef..988d05680a8 100644 --- a/indico/modules/events/persons/views.py +++ b/indico/modules/events/persons/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement diff --git a/indico/modules/events/posters.py b/indico/modules/events/posters.py index 18b0d8c62ab..93340766422 100644 --- a/indico/modules/events/posters.py +++ b/indico/modules/events/posters.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import division, unicode_literals - from collections import namedtuple from reportlab.lib.units import cm @@ -23,7 +21,7 @@ class PosterPDF(DesignerPDFBase): def __init__(self, template, config, event): - super(PosterPDF, self).__init__(template, config) + super().__init__(template, config) self.event = event def _build_config(self, config_data): @@ -55,7 +53,9 @@ def _build_pdf(self, canvas): self._draw_item(canvas, item, tpl_data, text, config.margin_horizontal, config.margin_vertical) def _draw_poster(self, canvas, registration, pos_x, pos_y): - """Draw a badge for a given registration, at position pos_x, pos_y (top-left corner). + """ + Draw a badge for a given registration, at position pos_x, + pos_y (top-left corner). """ config = self.config tpl_data = self.tpl_data diff --git a/indico/modules/events/registration/__init__.py b/indico/modules/events/registration/__init__.py index 01641df0aca..0f202c3f2bf 100644 --- a/indico/modules/events/registration/__init__.py +++ b/indico/modules/events/registration/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, render_template, session from indico.core import signals @@ -55,7 +53,7 @@ def _extend_event_management_menu(sender, event, **kwargs): @template_hook('conference-home-info') def _inject_regform_announcement(event, **kwargs): - from indico.modules.events.registration.util import get_registrations_with_tickets, get_event_regforms + from indico.modules.events.registration.util import get_event_regforms, get_registrations_with_tickets if event.has_feature('registration'): all_regforms = get_event_regforms(event, session.user) user_registrations = sum(regform[1] for regform in all_regforms) @@ -87,6 +85,8 @@ def _extend_event_menu(sender, **kwargs): def _visible_registration(event): if not event.has_feature('registration'): return False + if not event.can_access(session.user) and not (event.has_regform_in_acl and event.public_regform_access): + return False if any(form.is_scheduled for form in event.registration_forms): return True if not session.user: @@ -114,11 +114,11 @@ def _associate_registrations(user, **kwargs): subquery = db.session.query(reg_alias).filter(reg_alias.user_id == user.id, reg_alias.registration_form_id == Registration.registration_form_id, ~reg_alias.is_deleted) - registrations = (Registration - .find(Registration.user_id == None, # noqa - Registration.email.in_(user.all_emails), - ~subquery.exists(), - ~Registration.is_deleted) + registrations = (Registration.query + .filter(Registration.user_id.is_(None), + Registration.email.in_(user.all_emails), + ~subquery.exists(), + ~Registration.is_deleted) .order_by(Registration.submitted_dt.desc()) .all()) if not registrations: @@ -144,8 +144,9 @@ def _get_event_management_url(event, **kwargs): @signals.get_placeholders.connect_via('registration-invitation-email') def _get_invitation_placeholders(sender, invitation, **kwargs): - from indico.modules.events.registration.placeholders.invitations import (FirstNamePlaceholder, LastNamePlaceholder, - InvitationLinkPlaceholder) + from indico.modules.events.registration.placeholders.invitations import (FirstNamePlaceholder, + InvitationLinkPlaceholder, + LastNamePlaceholder) yield FirstNamePlaceholder yield LastNamePlaceholder yield InvitationLinkPlaceholder @@ -153,16 +154,18 @@ def _get_invitation_placeholders(sender, invitation, **kwargs): @signals.get_placeholders.connect_via('registration-email') def _get_registration_placeholders(sender, regform, registration, **kwargs): - from indico.modules.events.registration.placeholders.registrations import (IDPlaceholder, LastNamePlaceholder, - FirstNamePlaceholder, LinkPlaceholder, - EventTitlePlaceholder, - EventLinkPlaceholder, FieldPlaceholder) + from indico.modules.events.registration.placeholders.registrations import (EventLinkPlaceholder, + EventTitlePlaceholder, FieldPlaceholder, + FirstNamePlaceholder, IDPlaceholder, + LastNamePlaceholder, LinkPlaceholder, + RejectionReasonPlaceholder) yield FirstNamePlaceholder yield LastNamePlaceholder yield EventTitlePlaceholder yield EventLinkPlaceholder yield IDPlaceholder yield LinkPlaceholder + yield RejectionReasonPlaceholder yield FieldPlaceholder @@ -178,7 +181,7 @@ def _get_management_permissions(sender, **kwargs): @signals.event_management.get_cloners.connect def _get_registration_cloners(sender, **kwargs): - from indico.modules.events.registration.clone import RegistrationFormCloner, RegistrationCloner + from indico.modules.events.registration.clone import RegistrationCloner, RegistrationFormCloner yield RegistrationFormCloner yield RegistrationCloner diff --git a/indico/modules/events/registration/api.py b/indico/modules/events/registration/api.py index 22b866ee3b3..50a190f8151 100644 --- a/indico/modules/events/registration/api.py +++ b/indico/modules/events/registration/api.py @@ -1,36 +1,32 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - -from flask import jsonify, request +from flask import jsonify, request, session from sqlalchemy.orm import joinedload from werkzeug.exceptions import BadRequest, Forbidden from indico.core import signals from indico.modules.events.controllers.base import RHProtectedEventBase from indico.modules.events.models.events import Event +from indico.modules.events.registration.models.registrations import RegistrationState from indico.modules.events.registration.util import build_registration_api_data, build_registrations_api_data -from indico.modules.oauth import oauth -from indico.web.rh import RH +from indico.web.rh import RH, oauth_scope +@oauth_scope('registrants') class RHAPIRegistrant(RH): - """RESTful registrant API""" - - CSRF_ENABLED = False + """RESTful registrant API.""" - @oauth.require_oauth('registrants') def _check_access(self): - if not self.event.can_manage(request.oauth.user, permission='registration'): + if not self.event.can_manage(session.user, permission='registration'): raise Forbidden() def _process_args(self): - self.event = Event.find(id=request.view_args['event_id'], is_deleted=False).first_or_404() + self.event = Event.query.filter_by(id=request.view_args['event_id'], is_deleted=False).first_or_404() self._registration = (self.event.registrations .filter_by(id=request.view_args['registrant_id'], is_deleted=False) @@ -44,27 +40,29 @@ def _process_PATCH(self): if request.json is None: raise BadRequest('Expected JSON payload') - invalid_fields = request.json.viewkeys() - {'checked_in'} + invalid_fields = request.json.keys() - {'checked_in'} if invalid_fields: raise BadRequest("Invalid fields: {}".format(', '.join(invalid_fields))) if 'checked_in' in request.json: + if self._registration.state not in (RegistrationState.complete, RegistrationState.unpaid): + raise BadRequest('This registration cannot be marked as checked-in') self._registration.checked_in = bool(request.json['checked_in']) signals.event.registration_checkin_updated.send(self._registration) return jsonify(build_registration_api_data(self._registration)) +@oauth_scope('registrants') class RHAPIRegistrants(RH): - """RESTful registrants API""" + """RESTful registrants API.""" - @oauth.require_oauth('registrants') def _check_access(self): - if not self.event.can_manage(request.oauth.user, permission='registration'): + if not self.event.can_manage(session.user, permission='registration'): raise Forbidden() def _process_args(self): - self.event = Event.find(id=request.view_args['event_id'], is_deleted=False).first_or_404() + self.event = Event.query.filter_by(id=request.view_args['event_id'], is_deleted=False).first_or_404() def _process_GET(self): return jsonify(registrants=build_registrations_api_data(self.event)) diff --git a/indico/modules/events/registration/badges.py b/indico/modules/events/registration/badges.py index 28b51276444..70f27a95356 100644 --- a/indico/modules/events/registration/badges.py +++ b/indico/modules/events/registration/badges.py @@ -1,23 +1,19 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import division, unicode_literals - import re from collections import namedtuple -from itertools import izip, product +from itertools import product from reportlab.lib.units import cm from reportlab.lib.utils import ImageReader -from sqlalchemy.orm import subqueryload from werkzeug.exceptions import BadRequest from indico.modules.designer.pdf import DesignerPDFBase -from indico.modules.events.registration.models.registrations import Registration from indico.modules.events.registration.settings import DEFAULT_BADGE_SETTINGS from indico.util.i18n import _ from indico.util.placeholders import get_placeholders @@ -32,14 +28,9 @@ def _get_font_size(text): class RegistrantsListToBadgesPDF(DesignerPDFBase): - def __init__(self, template, config, event, registration_ids): - super(RegistrantsListToBadgesPDF, self).__init__(template, config) - self.registrations = (Registration.query.with_parent(event) - .filter(Registration.id.in_(registration_ids), - Registration.is_active) - .order_by(*Registration.order_by_name) - .options(subqueryload('data').joinedload('field_data')) - .all()) + def __init__(self, template, config, event, registrations): + super().__init__(template, config) + self.registrations = registrations def _build_config(self, config_data): return ConfigData(**config_data) @@ -49,7 +40,7 @@ def _iter_position(self, canvas, n_horizonal, n_vertical): config = self.config tpl_data = self.tpl_data while True: - for n_x, n_y in product(xrange(n_horizonal), xrange(n_vertical)): + for n_x, n_y in product(range(n_horizonal), range(n_vertical)): yield (config.left_margin + n_x * (tpl_data.width_cm + config.margin_columns), config.top_margin + n_y * (tpl_data.height_cm + config.margin_rows)) canvas.showPage() @@ -66,11 +57,14 @@ def _build_pdf(self, canvas): raise BadRequest(_('The template dimensions are too large for the page size you selected')) # Print a badge for each registration - for registration, (x, y) in izip(self.registrations, self._iter_position(canvas, n_horizontal, n_vertical)): + for registration, (x, y) in zip(self.registrations, self._iter_position(canvas, n_horizontal, n_vertical)): self._draw_badge(canvas, registration, self.template, self.tpl_data, x * cm, y * cm) def _draw_badge(self, canvas, registration, template, tpl_data, pos_x, pos_y): - """Draw a badge for a given registration, at position pos_x, pos_y (top-left corner).""" + """ + Draw a badge for a given registration, at position pos_x, + pos_y (top-left corner). + """ config = self.config badge_rect = (pos_x, self.height - pos_y - tpl_data.height_cm * cm, tpl_data.width_cm * cm, tpl_data.height_cm * cm) @@ -88,7 +82,7 @@ def _draw_badge(self, canvas, registration, template, tpl_data, pos_x, pos_y): placeholders = get_placeholders('designer-fields') # Print images first - image_placeholders = {name for name, placeholder in placeholders.viewitems() if placeholder.is_image} + image_placeholders = {name for name, placeholder in placeholders.items() if placeholder.is_image} items = sorted(tpl_data.items, key=lambda item: item['type'] not in image_placeholders) for item in items: @@ -113,7 +107,7 @@ def _build_pdf(self, canvas): n_horizontal = 1 n_vertical = 1 - for registration, (x, y) in izip(self.registrations, self._iter_position(canvas, n_horizontal, n_vertical)): + for registration, (x, y) in zip(self.registrations, self._iter_position(canvas, n_horizontal, n_vertical)): self._draw_badge(canvas, registration, self.template, self.tpl_data, x * cm, y * cm) if self.tpl_data.width > self.tpl_data.height: canvas.translate(self.width, self.height) @@ -163,7 +157,7 @@ def _build_pdf(self, canvas): if page_used: badges_mix += ([None] * (per_page - page_used)) + badges_mix[-page_used:] - positioned_badges = izip(badges_mix, self._iter_position(canvas, n_horizontal, n_vertical)) + positioned_badges = zip(badges_mix, self._iter_position(canvas, n_horizontal, n_vertical)) for i, (registration, (x, y)) in enumerate(positioned_badges): if registration is None: # blank item for an incomplete last page diff --git a/indico/modules/events/registration/blueprint.py b/indico/modules/events/registration/blueprint.py index 96dde9977b7..384fcd3d902 100644 --- a/indico/modules/events/registration/blueprint.py +++ b/indico/modules/events/registration/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.registration import api from indico.modules.events.registration.controllers import display from indico.modules.events.registration.controllers.compat import compat_registration @@ -16,7 +14,7 @@ from indico.web.flask.wrappers import IndicoBlueprint -_bp = IndicoBlueprint('event_registration', __name__, url_prefix='/event/', template_folder='templates', +_bp = IndicoBlueprint('event_registration', __name__, url_prefix='/event/', template_folder='templates', virtual_template_folder='events/registration', event_feature='registration') # Management @@ -70,7 +68,7 @@ _bp.add_url_rule('/manage/registration//registrations//approve', 'approve_registration', reglists.RHRegistrationApprove, methods=('POST',)) _bp.add_url_rule('/manage/registration//registrations//reject', - 'reject_registration', reglists.RHRegistrationReject, methods=('POST',)) + 'reject_registration', reglists.RHRegistrationReject, methods=('GET', 'POST')) _bp.add_url_rule('/manage/registration//registrations//reset', 'reset_registration', reglists.RHRegistrationReset, methods=('POST',)) _bp.add_url_rule('/manage/registration//registrations//withdraw', @@ -91,8 +89,10 @@ reglists.RHRegistrationsExportCSV, methods=('POST',)) _bp.add_url_rule('/manage/registration//registrations/registrations.xlsx', 'registrations_excel_export', reglists.RHRegistrationsExportExcel, methods=('POST',)) -_bp.add_url_rule('/manage/registration//registrations/modify-status', 'registrations_modify_status', - reglists.RHRegistrationsModifyStatus, methods=('POST',)) +_bp.add_url_rule('/manage/registration//registrations/approve', 'registrations_approve', + reglists.RHRegistrationsApprove, methods=('POST',)) +_bp.add_url_rule('/manage/registration//registrations/reject', 'registrations_reject', + reglists.RHRegistrationsReject, methods=('POST',)) _bp.add_url_rule('/manage/registration//registrations/check-in', 'registrations_check_in', reglists.RHRegistrationBulkCheckIn, methods=('POST',)) _bp.add_url_rule('/manage/registration//registrations/attachments', 'registrations_attachments_export', @@ -169,6 +169,8 @@ _bp.add_url_rule('/registrations//decline-invitation', 'decline_invitation', display.RHRegistrationFormDeclineInvitation, methods=('POST',)) _bp.add_url_rule('/registrations//ticket.pdf', 'ticket_download', display.RHTicketDownload) +_bp.add_url_rule('/registrations///avatar', 'registration_avatar', + display.RHRegistrationAvatar) # API @@ -180,7 +182,7 @@ # Participants -_bp_participation = IndicoBlueprint('event_participation', __name__, url_prefix='/event/', +_bp_participation = IndicoBlueprint('event_participation', __name__, url_prefix='/event/', template_folder='templates', virtual_template_folder='events/registration') _bp_participation.add_url_rule('/manage/participants/', 'manage', regforms.RHManageParticipants, methods=('GET', 'POST')) @@ -190,7 +192,6 @@ _compat_bp = IndicoBlueprint('compat_event_registration', __name__, url_prefix='/event/') _compat_bp.add_url_rule('/registration/', 'registration', compat_registration) _compat_bp.add_url_rule('/registration/', 'registration', compat_registration) -_compat_bp.add_url_rule('/registration/registrants', 'registrants', - make_compat_redirect_func(_bp, 'participant_list', view_args_conv={'event_id': 'confId'})) +_compat_bp.add_url_rule('/registration/registrants', 'registrants', make_compat_redirect_func(_bp, 'participant_list')) _compat_bp.add_url_rule('!/confRegistrantsDisplay.py/list', 'registrants_modpython', make_compat_redirect_func(_bp, 'participant_list')) diff --git a/indico/modules/events/registration/client/js/form/field.js b/indico/modules/events/registration/client/js/form/field.js index 0577b74d93a..86cf78ae76a 100644 --- a/indico/modules/events/registration/client/js/form/field.js +++ b/indico/modules/events/registration/client/js/form/field.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -13,7 +13,7 @@ ndRegForm.controller('FieldCtrl', function($scope, regFormFactory) { var getRequestParams = function(field) { return { - confId: $scope.confId, + eventId: $scope.eventId, sectionId: $scope.section.id, fieldId: field.id, confFormId: $scope.confFormId, diff --git a/indico/modules/events/registration/client/js/form/form.js b/indico/modules/events/registration/client/js/form/form.js index c4b6a85177a..1103820364f 100644 --- a/indico/modules/events/registration/client/js/form/form.js +++ b/indico/modules/events/registration/client/js/form/form.js @@ -1,10 +1,12 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. +/* eslint-disable import/unambiguous */ + window.ndRegForm = angular.module('nd.regform', ['ui.sortable', 'ngResource', 'ngSanitize']); // ============================================================================ @@ -13,12 +15,12 @@ window.ndRegForm = angular.module('nd.regform', ['ui.sortable', 'ngResource', 'n ndRegForm.value( 'editionURL', - Indico.Urls.Base + '/event/:confId/manage/registration/:confFormId/form/' + Indico.Urls.Base + '/event/:eventId/manage/registration/:confFormId/form/' ); ndRegForm.value( 'displayurl', - Indico.Urls.Base + '/event/:confId/registration/:confFormId/sections' + Indico.Urls.Base + '/event/:eventId/registration/:confFormId/sections' ); ndRegForm.value('sortableoptions', { @@ -84,7 +86,7 @@ ndRegForm.factory('regFormFactory', function( }, Sections: $resource( urls.section.add, - {confId: '@confId', sectionId: '@sectionId', confFormId: '@confFormId'}, + {eventId: '@eventId', sectionId: '@sectionId', confFormId: '@confFormId'}, { remove: {method: 'DELETE', url: urls.section.modify, isArray: true}, enable: {method: 'POST', url: urls.section.toggle, params: {enable: true}}, @@ -95,7 +97,7 @@ ndRegForm.factory('regFormFactory', function( ), Fields: $resource( urls.field.add, - {confId: '@confId', sectionId: '@sectionId', fieldId: '@fieldId', confFormId: '@confFormId'}, + {eventId: '@eventId', sectionId: '@sectionId', fieldId: '@fieldId', confFormId: '@confFormId'}, { remove: {method: 'DELETE', url: urls.field.modify}, enable: {method: 'POST', url: urls.field.toggle, params: {enable: true}}, @@ -106,7 +108,7 @@ ndRegForm.factory('regFormFactory', function( ), Labels: $resource( urls.text.add, - {confId: '@confId', sectionId: '@sectionId', fieldId: '@fieldId', confFormId: '@confFormId'}, + {eventId: '@eventId', sectionId: '@sectionId', fieldId: '@fieldId', confFormId: '@confFormId'}, { remove: {method: 'DELETE', url: urls.text.modify}, enable: {method: 'POST', url: urls.text.toggle, params: {enable: true}}, @@ -128,7 +130,7 @@ ndRegForm.directive('ndRegForm', function($rootScope, url, sortableoptions, regF templateUrl: url.tpl('registrationform.tpl.html'), scope: { - confId: '@', + eventId: '@', confFormId: '@', confCurrency: '@', confSections: '@', @@ -154,7 +156,7 @@ ndRegForm.directive('ndRegForm', function($rootScope, url, sortableoptions, regF $scope.registrationData = angular.fromJson($scope.registrationData); $scope.registrationMetaData = angular.fromJson($scope.registrationMetaData); - $rootScope.confId = $scope.confId; + $rootScope.eventId = $scope.eventId; $rootScope.confFormId = $scope.confFormId; $rootScope.eventStartDate = $scope.eventStartDate; $rootScope.eventEndDate = $scope.eventEndDate; @@ -192,7 +194,7 @@ ndRegForm.directive('ndRegForm', function($rootScope, url, sortableoptions, regF regFormFactory.Sections.save( { - confId: $scope.confId, + eventId: $scope.eventId, title: data.newsection.title, description: data.newsection.description, is_manager_only: managerOnly, @@ -216,7 +218,7 @@ ndRegForm.directive('ndRegForm', function($rootScope, url, sortableoptions, regF moveSection: function(section, position) { regFormFactory.Sections.move( { - confId: $scope.confId, + eventId: $scope.eventId, sectionId: section.id, endPos: position, confFormId: $scope.confFormId, @@ -234,7 +236,7 @@ ndRegForm.directive('ndRegForm', function($rootScope, url, sortableoptions, regF restoreSection: function(section) { regFormFactory.Sections.enable( { - confId: $rootScope.confId, + eventId: $rootScope.eventId, sectionId: section.id, confFormId: $rootScope.confFormId, }, @@ -258,7 +260,7 @@ ndRegForm.directive('ndRegForm', function($rootScope, url, sortableoptions, regF removeSection: function(section) { regFormFactory.Sections.remove( { - confId: $rootScope.confId, + eventId: $rootScope.eventId, sectionId: section.id, confFormId: $rootScope.confFormId, }, diff --git a/indico/modules/events/registration/client/js/form/section.js b/indico/modules/events/registration/client/js/form/section.js index f2d2c1aa699..24cd9de77ac 100644 --- a/indico/modules/events/registration/client/js/form/section.js +++ b/indico/modules/events/registration/client/js/form/section.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -11,7 +11,7 @@ ndRegForm.controller('SectionCtrl', function($scope, $rootScope, regFormFactory) var getRequestParams = function(section) { return { - confId: $rootScope.confId, + eventId: $rootScope.eventId, sectionId: section.id, confFormId: $rootScope.confFormId, }; diff --git a/indico/modules/events/registration/client/js/form/sectiontoolbar.js b/indico/modules/events/registration/client/js/form/sectiontoolbar.js index d4a413f5df1..b380a6dfd89 100644 --- a/indico/modules/events/registration/client/js/form/sectiontoolbar.js +++ b/indico/modules/events/registration/client/js/form/sectiontoolbar.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/registration/client/js/form/table.js b/indico/modules/events/registration/client/js/form/table.js index fb53c0c731e..cd7e5dbfbbb 100644 --- a/indico/modules/events/registration/client/js/form/table.js +++ b/indico/modules/events/registration/client/js/form/table.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/registration/client/js/form/templates.js b/indico/modules/events/registration/client/js/form/templates.js index 52e570c67f2..81615ebe450 100644 --- a/indico/modules/events/registration/client/js/form/templates.js +++ b/indico/modules/events/registration/client/js/form/templates.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/registration/client/js/form/tpls/registrationform.tpl.html b/indico/modules/events/registration/client/js/form/tpls/registrationform.tpl.html index cbebc08f132..22bdbde9858 100644 --- a/indico/modules/events/registration/client/js/form/tpls/registrationform.tpl.html +++ b/indico/modules/events/registration/client/js/form/tpls/registrationform.tpl.html @@ -67,7 +67,7 @@
- + { + const user = await showUserSearch({ + withExternalUsers: true, + single: true, + alwaysConfirm: true, + }); + if (user) { + const url = $('.js-add-user').data('href'); + location.href = build_url(url, {user}); + } }); $('.js-add-multiple-users').ajaxDialog({ dialogClasses: 'add-multiple-users-dialog', - onClose: function(data) { + onClose(data) { if (data) { $('.list-content').html(data.html); handleSelectedRowHighlight(true); diff --git a/indico/modules/events/registration/clone.py b/indico/modules/events/registration/clone.py index 2c3039fc9fd..0f059bd7a23 100644 --- a/indico/modules/events/registration/clone.py +++ b/indico/modules/events/registration/clone.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core import signals from indico.core.db import db from indico.core.db.sqlalchemy.util.models import get_simple_column_attrs @@ -57,7 +55,7 @@ def _has_content(self, event): return bool(event.registration_forms) def _clone_form_items(self, old_form, new_form, clone_all_revisions): - old_sections = RegistrationFormSection.find(RegistrationFormSection.registration_form_id == old_form.id) + old_sections = RegistrationFormSection.query.filter(RegistrationFormSection.registration_form_id == old_form.id) items_attrs = get_simple_column_attrs(RegistrationFormSection) for old_section in old_sections: new_section = RegistrationFormSection(**{attr: getattr(old_section, attr) for attr in items_attrs}) @@ -108,7 +106,7 @@ def has_conflicts(self, target_event): def run(self, new_event, cloners, shared_data, event_exists=False): form_map = shared_data['registration_forms']['form_map'] field_data_map = shared_data['registration_forms']['field_data_map'] - for old_form, new_form in form_map.iteritems(): + for old_form, new_form in form_map.items(): self._clone_registrations(old_form, new_form, field_data_map) self._synchronize_registration_friendly_id(new_event) db.session.flush() diff --git a/indico/modules/events/registration/clone_test.py b/indico/modules/events/registration/clone_test.py index a36ba0935b3..5701fa2c606 100644 --- a/indico/modules/events/registration/clone_test.py +++ b/indico/modules/events/registration/clone_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/registration/controllers/__init__.py b/indico/modules/events/registration/controllers/__init__.py index 40a201ccb6b..a3d9c4069a9 100644 --- a/indico/modules/events/registration/controllers/__init__.py +++ b/indico/modules/events/registration/controllers/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, redirect, request, session from sqlalchemy.orm import defaultload @@ -16,7 +14,7 @@ class RegistrationFormMixin: - """Mixin for single registration form RH""" + """Mixin for single registration form RH.""" normalize_url_spec = { 'locators': { diff --git a/indico/modules/events/registration/controllers/compat.py b/indico/modules/events/registration/controllers/compat.py index d496ea696af..3899b7143b8 100644 --- a/indico/modules/events/registration/controllers/compat.py +++ b/indico/modules/events/registration/controllers/compat.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import current_app, redirect, request from indico.modules.events.registration.models.legacy_mapping import LegacyRegistrationMapping @@ -16,15 +14,15 @@ @RHSimple.wrap_function def compat_registration(event_id, path=None): - url = url_for('event_registration.display_regform_list', confId=event_id) + url = url_for('event_registration.display_regform_list', event_id=event_id) try: registrant_id = int(request.args['registrantId']) authkey = request.args['authkey'] except KeyError: pass else: - mapping = (LegacyRegistrationMapping - .find(event_id=event_id, legacy_registrant_id=registrant_id, legacy_registrant_key=authkey) + mapping = (LegacyRegistrationMapping.query + .filter_by(event_id=event_id, legacy_registrant_id=registrant_id, legacy_registrant_key=authkey) .first()) if mapping: url = url_for('event_registration.display_regform', mapping.registration.locator.registrant) diff --git a/indico/modules/events/registration/controllers/display.py b/indico/modules/events/registration/controllers/display.py index 85c90b2d47d..3da8bfb31c8 100644 --- a/indico/modules/events/registration/controllers/display.py +++ b/indico/modules/events/registration/controllers/display.py @@ -1,17 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from operator import attrgetter from uuid import UUID from flask import flash, jsonify, redirect, request, session -from sqlalchemy.orm import contains_eager, subqueryload +from sqlalchemy.orm import contains_eager, joinedload, lazyload, load_only, subqueryload from werkzeug.exceptions import Forbidden, NotFound from indico.modules.auth.util import redirect_to_login @@ -30,6 +28,7 @@ from indico.modules.events.registration.views import (WPDisplayRegistrationFormConference, WPDisplayRegistrationFormSimpleEvent, WPDisplayRegistrationParticipantList) +from indico.modules.users.util import send_avatar, send_default_avatar from indico.util.fs import secure_filename from indico.util.i18n import _ from indico.web.flask.util import send_file, url_for @@ -71,7 +70,7 @@ def _check_restricted_event_access(self): class RHRegistrationFormRegistrationBase(RHRegistrationFormBase): - """Base for RHs handling individual registrations""" + """Base for RHs handling individual registrations.""" REGISTRATION_REQUIRED = True @@ -89,7 +88,7 @@ def _process_args(self): class RHRegistrationFormList(RHRegistrationFormDisplayBase): - """List of all registration forms in the event""" + """List of all registration forms in the event.""" ALLOW_PROTECTED_EVENT = True @@ -105,7 +104,7 @@ def _process(self): class RHParticipantList(RHRegistrationFormDisplayBase): - """List of all public registrations""" + """List of all public registrations.""" view_class = WPDisplayRegistrationParticipantList @@ -203,7 +202,7 @@ def _process(self): continue tables.append(self._participant_list_table(regform)) # There might be forms that have not been sorted by the user yet - tables += map(self._participant_list_table, regforms_dict.viewvalues()) + tables.extend(map(self._participant_list_table, regforms_dict.values())) published = (RegistrationForm.query.with_parent(self.event) .filter(RegistrationForm.publish_registrations_enabled) @@ -221,7 +220,7 @@ def _process(self): class InvitationMixin: - """Mixin for RHs that accept an invitation token""" + """Mixin for RHs that accept an invitation token.""" def _process_args(self): self.invitation = None @@ -234,13 +233,13 @@ def _process_args(self): except ValueError: flash(_("Your invitation code is not valid."), 'warning') return - self.invitation = RegistrationInvitation.find(uuid=token).with_parent(self.regform).first() + self.invitation = RegistrationInvitation.query.filter_by(uuid=token).with_parent(self.regform).first() if self.invitation is None: flash(_("This invitation does not exist or has been withdrawn."), 'warning') class RHRegistrationFormCheckEmail(RHRegistrationFormBase): - """Checks how an email will affect the registration""" + """Check how an email will affect the registration.""" ALLOW_PROTECTED_EVENT = True @@ -257,7 +256,7 @@ def _process(self): class RHRegistrationForm(InvitationMixin, RHRegistrationFormRegistrationBase): - """Display a registration form and registrations, and process submissions""" + """Display a registration form and registrations, and process submissions.""" REGISTRATION_REQUIRED = False ALLOW_PROTECTED_EVENT = True @@ -317,7 +316,7 @@ def _process(self): class RHRegistrationDisplayEdit(RegistrationEditMixin, RHRegistrationFormRegistrationBase): - """Submit a registration form""" + """Submit a registration form.""" template_file = 'display/registration_modify.html' management = False @@ -347,7 +346,7 @@ def success_url(self): class RHRegistrationWithdraw(RHRegistrationFormRegistrationBase): - """Withdraw a registration""" + """Withdraw a registration.""" def _check_access(self): RHRegistrationFormRegistrationBase._check_access(self) @@ -361,7 +360,7 @@ def _process(self): class RHRegistrationFormDeclineInvitation(InvitationMixin, RHRegistrationFormBase): - """Decline an invitation to register""" + """Decline an invitation to register.""" ALLOW_PROTECTED_EVENT = True @@ -377,7 +376,7 @@ def _process(self): class RHTicketDownload(RHRegistrationFormRegistrationBase): - """Generate ticket for a given registration""" + """Generate ticket for a given registration.""" def _check_access(self): RHRegistrationFormRegistrationBase._check_access(self) @@ -392,5 +391,34 @@ def _check_access(self): raise Forbidden def _process(self): - filename = secure_filename('{}-Ticket.pdf'.format(self.event.title), 'ticket.pdf') + filename = secure_filename(f'{self.event.title}-Ticket.pdf', 'ticket.pdf') return send_file(filename, generate_ticket(self.registration), 'application/pdf') + + +class RHRegistrationAvatar(RHDisplayEventBase): + """Display a standard avatar for a registration based on the full name.""" + + normalize_url_spec = { + 'locators': { + lambda self: self.registration + } + } + + def _process_args(self): + RHDisplayEventBase._process_args(self) + self.registration = (Registration.query + .filter(Registration.id == request.view_args['registration_id'], + ~Registration.is_deleted, + ~RegistrationForm.is_deleted) + .join(Registration.registration_form) + .options(load_only('id', 'registration_form_id', 'first_name', 'last_name'), + lazyload('*'), + joinedload('registration_form').load_only('id', 'event_id'), + joinedload('user').load_only('id', 'first_name', 'last_name', 'title', + 'picture_source', 'picture_metadata', 'picture')) + .one()) + + def _process(self): + if self.registration.user: + return send_avatar(self.registration.user) + return send_default_avatar(self.registration.full_name) diff --git a/indico/modules/events/registration/controllers/display_test.py b/indico/modules/events/registration/controllers/display_test.py index 79b674584af..e402f7c2b41 100644 --- a/indico/modules/events/registration/controllers/display_test.py +++ b/indico/modules/events/registration/controllers/display_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import pytest from flask import request, session @@ -24,7 +22,7 @@ def test_RHRegistrationForm_can_register(db, dummy_regform, dummy_reg, dummy_use invitation = RegistrationInvitation(registration_form=dummy_regform, email='foo@bar.com', first_name='foo', last_name='bar', affiliation='test') db.session.flush() - request.view_args = {'reg_form_id': dummy_regform.id, 'confId': dummy_regform.event_id} + request.view_args = {'reg_form_id': dummy_regform.id, 'event_id': dummy_regform.event_id} request.args = {'invitation': invitation.uuid} rh = RHRegistrationForm() rh._process_args() @@ -33,13 +31,13 @@ def test_RHRegistrationForm_can_register(db, dummy_regform, dummy_reg, dummy_use assert not rh._can_register() # not open dummy_regform.start_dt = now_utc(False) assert rh._can_register() - session.user = dummy_user # registered in dummy_reg + session.set_session_user(dummy_user) # registered in dummy_reg assert not rh._can_register() dummy_reg.state = RegistrationState.rejected assert not rh._can_register() # being rejected does not allow registering again dummy_reg.state = RegistrationState.withdrawn assert not rh._can_register() # being withdrawn does not allow registering again - session.user = create_user(123, email='user@example.com') + session.set_session_user(create_user(123, email='user@example.com')) assert rh._can_register() dummy_regform.registration_limit = 1 assert rh._can_register() # withdrawn/rejected do not count against limit diff --git a/indico/modules/events/registration/controllers/management/__init__.py b/indico/modules/events/registration/controllers/management/__init__.py index b8f6d7e0b2f..8e2d699a9e7 100644 --- a/indico/modules/events/registration/controllers/management/__init__.py +++ b/indico/modules/events/registration/controllers/management/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request from sqlalchemy.orm import contains_eager, defaultload @@ -18,13 +16,13 @@ class RHManageRegFormsBase(RHManageEventBase): - """Base class for all registration management RHs""" + """Base class for all registration management RHs.""" PERMISSION = 'registration' class RHManageRegFormBase(RegistrationFormMixin, RHManageRegFormsBase): - """Base class for a specific registration form""" + """Base class for a specific registration form.""" def _process_args(self): RHManageRegFormsBase._process_args(self) @@ -33,7 +31,7 @@ def _process_args(self): class RHManageRegistrationBase(RHManageRegFormBase): - """Base class for a specific registration""" + """Base class for a specific registration.""" normalize_url_spec = { 'locators': { @@ -43,10 +41,10 @@ class RHManageRegistrationBase(RHManageRegFormBase): def _process_args(self): RHManageRegFormBase._process_args(self) - self.registration = (Registration - .find(Registration.id == request.view_args['registration_id'], - ~Registration.is_deleted, - ~RegistrationForm.is_deleted) + self.registration = (Registration.query + .filter(Registration.id == request.view_args['registration_id'], + ~Registration.is_deleted, + ~RegistrationForm.is_deleted) .join(Registration.registration_form) .options(contains_eager(Registration.registration_form) .defaultload('form_items') diff --git a/indico/modules/events/registration/controllers/management/fields.py b/indico/modules/events/registration/controllers/management/fields.py index 9bfb8d4b488..5e10eaddc21 100644 --- a/indico/modules/events/registration/controllers/management/fields.py +++ b/indico/modules/events/registration/controllers/management/fields.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import jsonify, request, session from werkzeug.exceptions import BadRequest @@ -34,7 +32,7 @@ def _fill_form_field_with_data(field, field_data, set_data=True): class RHManageRegFormFieldBase(RHManageRegFormSectionBase): - """Base class for a specific field within a registration form""" + """Base class for a specific field within a registration form.""" field_class = RegistrationFormField normalize_url_spec = { @@ -49,7 +47,7 @@ def _process_args(self): class RHRegistrationFormToggleFieldState(RHManageRegFormFieldBase): - """Enable/Disable a field""" + """Enable/Disable a field.""" def _process(self): enabled = request.args.get('enable') == 'true' @@ -64,7 +62,7 @@ def _process(self): class RHRegistrationFormModifyField(RHManageRegFormFieldBase): - """Remove/Modify a field""" + """Remove/Modify a field.""" def _process_DELETE(self): if self.field.type == RegistrationFormItemType.field_pd: @@ -87,7 +85,7 @@ def _process_PATCH(self): class RHRegistrationFormMoveField(RHManageRegFormFieldBase): - """Change position of a field within the section""" + """Change position of a field within the section.""" def _process(self): new_position = request.json['endPos'] + 1 @@ -104,7 +102,7 @@ def fn(field): return (old_position < field.position <= new_position and field.id != self.field.id and not field.is_deleted and field.is_enabled) start_enum = self.field.position - to_update = filter(fn, self.section.children) + to_update = list(filter(fn, self.section.children)) self.field.position = new_position for pos, field in enumerate(to_update, start_enum): field.position = pos @@ -113,7 +111,7 @@ def fn(field): class RHRegistrationFormAddField(RHManageRegFormSectionBase): - """Add a field to the section""" + """Add a field to the section.""" def _process(self): field_data = snakify_keys(request.json['fieldData']) @@ -125,12 +123,12 @@ def _process(self): class RHRegistrationFormToggleTextState(RHRegistrationFormToggleFieldState): - """Enable/Disable a static text field""" + """Enable/Disable a static text field.""" field_class = RegistrationFormText class RHRegistrationFormModifyText(RHRegistrationFormModifyField): - """Remove/Modify a static text field""" + """Remove/Modify a static text field.""" field_class = RegistrationFormText def _process_PATCH(self): @@ -141,12 +139,12 @@ def _process_PATCH(self): class RHRegistrationFormMoveText(RHRegistrationFormMoveField): - """Change position of a static text field within the section""" + """Change position of a static text field within the section.""" field_class = RegistrationFormText class RHRegistrationFormAddText(RHManageRegFormSectionBase): - """Add a static text field to a section""" + """Add a static text field to a section.""" def _process(self): field_data = snakify_keys(request.json['fieldData']) diff --git a/indico/modules/events/registration/controllers/management/invitations.py b/indico/modules/events/registration/controllers/management/invitations.py index 3be697bd741..a64673a1da9 100644 --- a/indico/modules/events/registration/controllers/management/invitations.py +++ b/indico/modules/events/registration/controllers/management/invitations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, request from sqlalchemy.orm import joinedload @@ -38,7 +36,7 @@ def _render_invitation_list(regform): class RHRegistrationFormInvitations(RHManageRegFormBase): - """Overview of all registration invitations""" + """Overview of all registration invitations.""" def _process(self): invitations = _query_invitation_list(self.regform) @@ -47,7 +45,7 @@ def _process(self): class RHRegistrationFormInvite(RHManageRegFormBase): - """Invite someone to register""" + """Invite someone to register.""" def _create_invitation(self, user, skip_moderation, email_from, email_subject, email_body): invitation = RegistrationInvitation( @@ -94,7 +92,7 @@ def _process_args(self): class RHRegistrationFormDeleteInvitation(RHRegistrationFormInvitationBase): - """Delete a registration invitation""" + """Delete a registration invitation.""" def _process(self): db.session.delete(self.invitation) diff --git a/indico/modules/events/registration/controllers/management/regforms.py b/indico/modules/events/registration/controllers/management/regforms.py index 8e459cfd455..3ce4bd4b638 100644 --- a/indico/modules/events/registration/controllers/management/regforms.py +++ b/indico/modules/events/registration/controllers/management/regforms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from operator import attrgetter, itemgetter from flask import flash, redirect, session @@ -38,7 +36,7 @@ class RHManageRegistrationForms(RHManageRegFormsBase): - """List all registrations forms for an event""" + """List all registrations forms for an event.""" def _process(self): regforms = (RegistrationForm.query @@ -49,7 +47,7 @@ def _process(self): class RHManageRegistrationFormsDisplay(RHManageRegFormsBase): - """Customize the display of registrations on the public page""" + """Customize the display of registrations on the public page.""" def _process(self): regforms = sorted(self.event.registration_forms, key=lambda f: f.title.lower()) @@ -71,7 +69,7 @@ def _process(self): if column_name in available_columns: enabled_columns.append({'id': column_name, 'title': available_columns[column_name]}) del available_columns[column_name] - for column_name, column_title in available_columns.iteritems(): + for column_name, column_title in available_columns.items(): disabled_columns.append({'id': column_name, 'title': column_title}) disabled_columns.sort(key=itemgetter('title')) @@ -88,7 +86,7 @@ def _process(self): if regform.publish_registrations_enabled: enabled_forms.append(regform) del available_forms[form_id] - for form_id, regform in available_forms.iteritems(): + for form_id, regform in available_forms.items(): # There might be forms with publication enabled that haven't been sorted by the user yet. if regform.publish_registrations_enabled: enabled_forms.append(regform) @@ -104,7 +102,10 @@ def _process(self): class RHManageRegistrationFormDisplay(RHManageRegFormBase): - """Choose the columns to be shown on the participant list for a particular form""" + """ + Choose the columns to be shown on the participant list for + a particular form. + """ def _process(self): form = ParticipantsDisplayFormColumnsForm() @@ -123,13 +124,13 @@ def _process(self): enabled_fields.append(field) del available_fields[field_id] - disabled_fields = available_fields.values() + disabled_fields = list(available_fields.values()) return jsonify_template('events/registration/management/regform_display_form_columns.html', form=form, enabled_columns=enabled_fields, disabled_columns=disabled_fields) class RHManageParticipants(RHManageRegFormsBase): - """Show and enable the dummy registration form for participants""" + """Show and enable the dummy registration form for participants.""" def _process_POST(self): regform = self.event.participation_regform @@ -142,7 +143,7 @@ def _process_POST(self): db.session.flush() signals.event.registration_form_created.send(regform) self.event.log(EventLogRealm.management, EventLogKind.positive, 'Registration', - 'Registration form "{}" has been created'.format(regform.title), session.user) + f'Registration form "{regform.title}" has been created', session.user) return redirect(url_for('event_registration.manage_regform', regform)) def _process_GET(self): @@ -155,7 +156,7 @@ def _process_GET(self): class RHRegistrationFormCreate(RHManageRegFormsBase): - """Creates a new registration form""" + """Create a new registration form.""" def _process(self): form = RegistrationFormForm(event=self.event, @@ -169,21 +170,21 @@ def _process(self): signals.event.registration_form_created.send(regform) flash(_('Registration form has been successfully created'), 'success') self.event.log(EventLogRealm.management, EventLogKind.positive, 'Registration', - 'Registration form "{}" has been created'.format(regform.title), session.user) + f'Registration form "{regform.title}" has been created', session.user) return redirect(url_for('.manage_regform', regform)) return WPManageRegistration.render_template('management/regform_edit.html', self.event, form=form, regform=None) class RHRegistrationFormManage(RHManageRegFormBase): - """Specific registration form management""" + """Specific registration form management.""" def _process(self): return WPManageRegistration.render_template('management/regform.html', self.event, regform=self.regform) class RHRegistrationFormEdit(RHManageRegFormBase): - """Edit a registration form""" + """Edit a registration form.""" def _get_form_defaults(self): return FormDefaults(self.regform, limit_registrations=self.regform.registration_limit is not None) @@ -201,7 +202,7 @@ def _process(self): class RHRegistrationFormDelete(RHManageRegFormBase): - """Delete a registration form""" + """Delete a registration form.""" def _process(self): self.regform.is_deleted = True @@ -212,7 +213,7 @@ def _process(self): class RHRegistrationFormOpen(RHManageRegFormBase): - """Opens registration for a registration form""" + """Open registration for a registration form.""" def _process(self): old_dts = (self.regform.start_dt, self.regform.end_dt) @@ -225,15 +226,15 @@ def _process(self): new_dts = (self.regform.start_dt, self.regform.end_dt) if new_dts != old_dts: if not old_dts[1]: - log_text = 'Registration form "{}" was opened'.format(self.regform.title) + log_text = f'Registration form "{self.regform.title}" was opened' else: - log_text = 'Registration form "{}" was reopened'.format(self.regform.title) + log_text = f'Registration form "{self.regform.title}" was reopened' self.event.log(EventLogRealm.event, EventLogKind.change, 'Registration', log_text, session.user) return redirect(url_for('.manage_regform', self.regform)) class RHRegistrationFormClose(RHManageRegFormBase): - """Closes registrations for a registration form""" + """Close registrations for a registration form.""" def _process(self): self.regform.end_dt = now_utc() @@ -241,13 +242,13 @@ def _process(self): self.regform.start_dt = self.regform.end_dt flash(_("Registrations for {} are now closed").format(self.regform.title), 'success') logger.info("Registrations for %s closed by %s", self.regform, session.user) - log_text = 'Registration form "{}" was closed'.format(self.regform.title) + log_text = f'Registration form "{self.regform.title}" was closed' self.event.log(EventLogRealm.event, EventLogKind.change, 'Registration', log_text, session.user) return redirect(url_for('.manage_regform', self.regform)) class RHRegistrationFormSchedule(RHManageRegFormBase): - """Schedules registrations for a registration form""" + """Schedule registrations for a registration form.""" def _process(self): form = RegistrationFormScheduleForm(obj=FormDefaults(self.regform), regform=self.regform) @@ -262,7 +263,7 @@ def _process(self): class RHRegistrationFormModify(RHManageRegFormBase): - """Modify the form of a registration form""" + """Modify the form of a registration form.""" def _process(self): return WPManageRegistration.render_template('management/regform_modify.html', self.event, @@ -271,7 +272,7 @@ def _process(self): class RHRegistrationFormStats(RHManageRegFormBase): - """Display registration form stats page""" + """Display registration form stats page.""" def _process(self): regform_stats = [OverviewStats(self.regform)] @@ -281,7 +282,7 @@ def _process(self): class RHManageRegistrationManagers(RHManageRegFormsBase): - """Modify event managers with registration role""" + """Modify event managers with registration role.""" def _process(self): reg_managers = {p.principal for p in self.event.acl_entries diff --git a/indico/modules/events/registration/controllers/management/reglists.py b/indico/modules/events/registration/controllers/management/reglists.py index 204142dfb2d..001f5064faf 100644 --- a/indico/modules/events/registration/controllers/management/reglists.py +++ b/indico/modules/events/registration/controllers/management/reglists.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import uuid from io import BytesIO @@ -16,11 +14,11 @@ from werkzeug.exceptions import BadRequest, Forbidden, NotFound from indico.core import signals +from indico.core.cache import make_scoped_cache from indico.core.config import config from indico.core.db import db from indico.core.errors import NoReportError from indico.core.notifications import make_email, send_email -from indico.legacy.common.cache import GenericCache from indico.legacy.pdfinterface.conference import RegistrantsListToBookPDF, RegistrantsListToPDF from indico.modules.designer import PageLayout, TemplateType from indico.modules.designer.models.templates import DesignerTemplate @@ -36,7 +34,8 @@ from indico.modules.events.registration.controllers.management import (RHManageRegFormBase, RHManageRegFormsBase, RHManageRegistrationBase) from indico.modules.events.registration.forms import (BadgeSettingsForm, CreateMultipleRegistrationsForm, - EmailRegistrantsForm, ImportRegistrationsForm) + EmailRegistrantsForm, ImportRegistrationsForm, + RejectRegistrantsForm) from indico.modules.events.registration.models.items import PersonalDataType, RegistrationFormItemType from indico.modules.events.registration.models.registrations import Registration, RegistrationData, RegistrationState from indico.modules.events.registration.notifications import notify_registration_state_update @@ -46,17 +45,18 @@ import_registrations_from_csv, make_registration_form) from indico.modules.events.registration.views import WPManageRegistration from indico.modules.events.util import ZipGeneratorMixin -from indico.modules.users import User from indico.util.fs import secure_filename from indico.util.i18n import _, ngettext +from indico.util.marshmallow import Principal from indico.util.placeholders import replace_placeholders from indico.util.spreadsheets import send_csv, send_xlsx +from indico.web.args import use_kwargs from indico.web.flask.templating import get_template_module from indico.web.flask.util import send_file, url_for -from indico.web.util import jsonify_data, jsonify_template +from indico.web.util import jsonify_data, jsonify_form, jsonify_template -badge_cache = GenericCache('badge-printing') +badge_cache = make_scoped_cache('badge-printing') def _render_registration_details(registration): @@ -66,7 +66,7 @@ def _render_registration_details(registration): class RHRegistrationsListManage(RHManageRegFormBase): - """List all registrations of a specific registration form of an event""" + """List all registrations of a specific registration form of an event.""" def _process(self): if self.list_generator.static_link_used: @@ -81,7 +81,7 @@ def _process(self): class RHRegistrationsListCustomize(RHManageRegFormBase): - """Filter options and columns to display for a registrations list of an event""" + """Filter options and columns to display for a registrations list of an event.""" ALLOW_LOCKED = True @@ -100,7 +100,7 @@ def _process_POST(self): class RHRegistrationListStaticURL(RHManageRegFormBase): - """Generate a static URL for the configuration of the registrations list""" + """Generate a static URL for the configuration of the registrations list.""" ALLOW_LOCKED = True @@ -109,7 +109,7 @@ def _process(self): class RHRegistrationDetails(RHManageRegistrationBase): - """Displays information about a registration""" + """Display information about a registration.""" def _process(self): registration_details_html = _render_registration_details(self.registration) @@ -119,7 +119,7 @@ def _process(self): class RHRegistrationDownloadAttachment(RHManageRegFormsBase): - """Download a file attached to a registration""" + """Download a file attached to a registration.""" normalize_url_spec = { 'locators': { @@ -129,10 +129,10 @@ class RHRegistrationDownloadAttachment(RHManageRegFormsBase): def _process_args(self): RHManageRegFormsBase._process_args(self) - self.field_data = (RegistrationData - .find(RegistrationData.registration_id == request.view_args['registration_id'], - RegistrationData.field_data_id == request.view_args['field_data_id'], - RegistrationData.filename.isnot(None)) + self.field_data = (RegistrationData.query + .filter(RegistrationData.registration_id == request.view_args['registration_id'], + RegistrationData.field_data_id == request.view_args['field_data_id'], + RegistrationData.filename.isnot(None)) .options(joinedload('registration').joinedload('registration_form')) .one()) @@ -159,7 +159,7 @@ def success_url(self): class RHRegistrationsActionBase(RHManageRegFormBase): - """Base class for classes performing actions on registrations""" + """Base class for classes performing actions on registrations.""" registration_query_options = () @@ -175,7 +175,7 @@ def _process_args(self): class RHRegistrationEmailRegistrantsPreview(RHRegistrationsActionBase): - """Previews the email that will be sent to registrants""" + """Preview the email that will be sent to registrants.""" def _process(self): if not self.registrations: @@ -193,7 +193,7 @@ def _process(self): class RHRegistrationEmailRegistrants(RHRegistrationsActionBase): - """Send email to selected registrants""" + """Send email to selected registrants.""" def _send_emails(self, form): for registration in self.registrations: @@ -233,7 +233,7 @@ def _process(self): class RHRegistrationDelete(RHRegistrationsActionBase): - """Delete selected registrations""" + """Delete selected registrations.""" def _process(self): for registration in self.registrations: @@ -241,7 +241,7 @@ def _process(self): signals.event.registration_deleted.send(registration) logger.info('Registration %s deleted by %s', registration, session.user) registration.log(EventLogRealm.management, EventLogKind.negative, 'Registration', - 'Registration deleted: {}'.format(registration.full_name), + f'Registration deleted: {registration.full_name}', session.user, data={'Email': registration.email}) num_reg_deleted = len(self.registrations) flash(ngettext("Registration was deleted.", @@ -250,20 +250,15 @@ def _process(self): class RHRegistrationCreate(RHManageRegFormBase): - """Create new registration (management area)""" + """Create new registration (management area).""" - def _get_user_data(self): - user_id = request.args.get('user') - if user_id is None: + @use_kwargs({ + 'user': Principal(allow_external_users=True, missing=None), + }, location='query') + def _get_user_data(self, user): + if user is None: return {} - elif user_id.isdigit(): - # existing indico user - user = User.find_first(id=user_id, is_deleted=False) - user_data = {t.name: getattr(user, t.name, None) if user else '' for t in PersonalDataType} - else: - # non-indico user - data = GenericCache('pending_identities').get(user_id, {}) - user_data = {t.name: data.get(t.name) for t in PersonalDataType} + user_data = {t.name: getattr(user, t.name, None) for t in PersonalDataType} user_data['title'] = get_title_uuid(self.regform, user_data['title']) return user_data @@ -286,7 +281,7 @@ def _process(self): class RHRegistrationCreateMultiple(RHManageRegFormBase): - """Create multiple registrations for Indico users (management area)""" + """Create multiple registrations for Indico users (management area).""" def _register_user(self, user, notify): # Fill only the personal data fields, custom fields are left empty. @@ -309,7 +304,7 @@ def _process(self): class RHRegistrationsExportBase(RHRegistrationsActionBase): - """Base class for all registration list export RHs""" + """Base class for all registration list export RHs.""" ALLOW_LOCKED = True registration_query_options = (subqueryload('data'),) @@ -320,7 +315,7 @@ def _process_args(self): class RHRegistrationsExportPDFTable(RHRegistrationsExportBase): - """Export registration list to a PDF in table style""" + """Export registration list to a PDF in table style.""" def _process(self): pdf = RegistrantsListToPDF(self.event, reglist=self.registrations, display=self.export_config['regform_items'], @@ -336,7 +331,7 @@ def _process(self): class RHRegistrationsExportPDFBook(RHRegistrationsExportBase): - """Export registration list to a PDF in book style""" + """Export registration list to a PDF in book style.""" def _process(self): static_item_ids, item_ids = self.list_generator.get_item_ids() @@ -345,7 +340,7 @@ def _process(self): class RHRegistrationsExportCSV(RHRegistrationsExportBase): - """Export registration list to a CSV file""" + """Export registration list to a CSV file.""" def _process(self): headers, rows = generate_spreadsheet_from_registrations(self.registrations, self.export_config['regform_items'], @@ -354,7 +349,7 @@ def _process(self): class RHRegistrationsExportExcel(RHRegistrationsExportBase): - """Export registration list to an XLSX file""" + """Export registration list to an XLSX file.""" def _process(self): headers, rows = generate_spreadsheet_from_registrations(self.registrations, self.export_config['regform_items'], @@ -414,15 +409,20 @@ def _process(self): else: pdf_class = RegistrantsListToBadgesPDF registration_ids = config_params.pop('registration_ids') - registrations = Registration.query.filter(Registration.id.in_(registration_ids)).all() + registrations = (Registration.query.with_parent(self.event) + .filter(Registration.id.in_(registration_ids), + Registration.is_active) + .order_by(*Registration.order_by_name) + .options(subqueryload('data').joinedload('field_data')) + .all()) signals.event.designer.print_badge_template.send(self.template, regform=self.regform, registrations=registrations) - pdf = pdf_class(self.template, config_params, self.event, registration_ids) - return send_file('Badges-{}.pdf'.format(self.event.id), pdf.get_pdf(), 'application/pdf') + pdf = pdf_class(self.template, config_params, self.event, registrations) + return send_file(f'Badges-{self.event.id}.pdf', pdf.get_pdf(), 'application/pdf') class RHRegistrationsConfigBadges(RHRegistrationsActionBase): - """Print badges for the selected registrations""" + """Print badges for the selected registrations.""" ALLOW_LOCKED = True TICKET_BADGES = False @@ -439,7 +439,7 @@ class RHRegistrationsConfigBadges(RHRegistrationsActionBase): 'A8': (5.2, 7.4), } - format_map_landscape = {name: (h, w) for name, (w, h) in format_map_portrait.iteritems()} + format_map_landscape = {name: (h, w) for name, (w, h) in format_map_portrait.items()} def _process_args(self): RHManageRegFormBase._process_args(self) @@ -454,7 +454,7 @@ def _process_args(self): def _get_format(self, tpl): from indico.modules.designer.pdf import PIXELS_CM format_map = self.format_map_landscape if tpl.data['width'] > tpl.data['height'] else self.format_map_portrait - return next((frm for frm, frm_size in format_map.iteritems() + return next((frm for frm, frm_size in format_map.items() if (frm_size[0] == float(tpl.data['width']) / PIXELS_CM and frm_size[1] == float(tpl.data['height']) / PIXELS_CM)), 'custom') @@ -496,8 +496,8 @@ def _process(self): event_badge_settings.set_multi(self.event, data) data['registration_ids'] = [x.id for x in registrations] - key = unicode(uuid.uuid4()) - badge_cache.set(key, data, time=1800) + key = str(uuid.uuid4()) + badge_cache.set(key, data, timeout=1800) download_url = url_for('.registrations_print_badges', self.regform, template_id=template_id, uuid=key) return jsonify_data(flash=False, redirect=download_url, redirect_no_loading=True) return jsonify_template('events/registration/management/print_badges.html', event=self.event, @@ -506,20 +506,20 @@ def _process(self): class RHRegistrationsConfigTickets(RHRegistrationsConfigBadges): - """Print tickets for selected registrations""" + """Print tickets for selected registrations.""" TICKET_BADGES = True @property def _default_template_id(self): - return unicode(self.regform.ticket_template_id) if self.regform.ticket_template_id else None + return str(self.regform.ticket_template_id) if self.regform.ticket_template_id else None def _filter_registrations(self, registrations): return [r for r in registrations if not r.is_ticket_blocked] class RHRegistrationTogglePayment(RHManageRegistrationBase): - """Modify the payment status of a registration""" + """Modify the payment status of a registration.""" def _process(self): pay = request.form.get('pay') == '1' @@ -536,21 +536,22 @@ def _process(self): return jsonify_data(html=_render_registration_details(self.registration)) -def _modify_registration_status(registration, approve): +def _modify_registration_status(registration, approve, rejection_reason='', attach_rejection_reason=False): if registration.state != RegistrationState.pending: return if approve: registration.update_state(approved=True) else: + registration.rejection_reason = rejection_reason registration.update_state(rejected=True) db.session.flush() - notify_registration_state_update(registration) + notify_registration_state_update(registration, attach_rejection_reason) status = 'approved' if approve else 'rejected' logger.info('Registration %s was %s by %s', registration, status, session.user) class RHRegistrationApprove(RHManageRegistrationBase): - """Accept a registration""" + """Accept a registration.""" def _process(self): _modify_registration_status(self.registration, approve=True) @@ -558,15 +559,20 @@ def _process(self): class RHRegistrationReject(RHManageRegistrationBase): - """Reject a registration""" + """Reject a registration.""" def _process(self): - _modify_registration_status(self.registration, approve=False) - return jsonify_data(html=_render_registration_details(self.registration)) + form = RejectRegistrantsForm() + message = _("Rejecting this registration will trigger a notification email.") + if form.validate_on_submit(): + _modify_registration_status(self.registration, approve=False, rejection_reason=form.rejection_reason.data, + attach_rejection_reason=form.attach_rejection_reason.data) + return jsonify_data(html=_render_registration_details(self.registration)) + return jsonify_form(form, disabled_until_change=False, submit=_('Reject'), message=message) class RHRegistrationReset(RHManageRegistrationBase): - """Reset a registration back to a non-approved status""" + """Reset a registration back to a non-approved status.""" def _process(self): if self.registration.has_conflict(): @@ -575,12 +581,14 @@ def _process(self): if self.registration.state in (RegistrationState.complete, RegistrationState.unpaid): self.registration.update_state(approved=False) elif self.registration.state == RegistrationState.rejected: + self.registration.rejection_reason = '' self.registration.update_state(rejected=False) elif self.registration.state == RegistrationState.withdrawn: self.registration.update_state(withdrawn=False) notify_registration_state_update(self.registration) else: raise BadRequest(_('The registration cannot be reset in its current state.')) + self.registration.checked_in = False logger.info('Registration %r was reset by %r', self.registration, session.user) return jsonify_data(html=_render_registration_details(self.registration)) @@ -599,9 +607,11 @@ def _process(self): class RHRegistrationCheckIn(RHManageRegistrationBase): - """Set checked in state of a registration""" + """Set checked in state of a registration.""" def _process_PUT(self): + if self.registration.state not in (RegistrationState.complete, RegistrationState.unpaid): + raise BadRequest(_('This registration cannot be marked as checked-in')) self.registration.checked_in = True signals.event.registration_checkin_updated.send(self.registration) return jsonify_data(html=_render_registration_details(self.registration)) @@ -613,12 +623,14 @@ def _process_DELETE(self): class RHRegistrationBulkCheckIn(RHRegistrationsActionBase): - """Bulk apply check-in/not checked-in state to registrations""" + """Bulk apply check-in/not checked-in state to registrations.""" def _process(self): check_in = request.form['flag'] == '1' msg = 'checked-in' if check_in else 'not checked-in' for registration in self.registrations: + if registration.state not in (RegistrationState.complete, RegistrationState.unpaid): + continue registration.checked_in = check_in signals.event.registration_checkin_updated.send(registration) logger.info('Registration %s marked as %s by %s', registration, msg, session.user) @@ -626,33 +638,46 @@ def _process(self): return jsonify_data(**self.list_generator.render_list()) -class RHRegistrationsModifyStatus(RHRegistrationsActionBase): - """Accept/Reject selected registrations""" +class RHRegistrationsApprove(RHRegistrationsActionBase): + """Accept selected registrations from registration list.""" def _process(self): - approve = request.form['flag'] == '1' for registration in self.registrations: - _modify_registration_status(registration, approve) - flash(_("The status of the selected registrations was updated successfully."), 'success') + _modify_registration_status(registration, approve=True) + flash(_("The selected registrations were successfully approved."), 'success') return jsonify_data(**self.list_generator.render_list()) +class RHRegistrationsReject(RHRegistrationsActionBase): + """Reject selected registrations from registration list.""" + + def _process(self): + form = RejectRegistrantsForm(registration_id=[r.id for r in self.registrations]) + message = _("Rejecting these registrations will trigger a notification email for each registrant.") + if form.validate_on_submit(): + for registration in self.registrations: + _modify_registration_status(registration, approve=False, rejection_reason=form.rejection_reason.data, + attach_rejection_reason=form.attach_rejection_reason.data) + flash(_("The selected registrations were successfully rejected."), 'success') + return jsonify_data(**self.list_generator.render_list()) + return jsonify_form(form, disabled_until_change=False, submit=_('Reject'), message=message) + + class RHRegistrationsExportAttachments(RHRegistrationsExportBase, ZipGeneratorMixin): - """Export registration attachments in a zip file""" + """Export registration attachments in a zip file.""" def _prepare_folder_structure(self, attachment): registration = attachment.registration regform_title = secure_filename(attachment.registration.registration_form.title, 'registration_form') registrant_name = secure_filename("{}_{}".format(registration.get_full_name(), - unicode(registration.friendly_id)), registration.friendly_id) + str(registration.friendly_id)), registration.friendly_id) file_name = secure_filename("{}_{}_{}".format(attachment.field_data.field.title, attachment.field_data.field_id, attachment.filename), attachment.filename) return os.path.join(*self._adjust_path_length([regform_title, registrant_name, file_name])) def _iter_items(self, attachments): - for reg_attachments in attachments.itervalues(): - for reg_attachment in reg_attachments: - yield reg_attachment + for reg_attachments in attachments.values(): + yield from reg_attachments def _process(self): attachments = {} diff --git a/indico/modules/events/registration/controllers/management/sections.py b/indico/modules/events/registration/controllers/management/sections.py index 78397ab8834..01ed27f96bf 100644 --- a/indico/modules/events/registration/controllers/management/sections.py +++ b/indico/modules/events/registration/controllers/management/sections.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import jsonify, request, session from werkzeug.exceptions import BadRequest @@ -19,7 +17,7 @@ class RHManageRegFormSectionBase(RHManageRegFormBase): - """Base class for a specific registration form section""" + """Base class for a specific registration form section.""" normalize_url_spec = { 'locators': { @@ -33,7 +31,7 @@ def _process_args(self): class RHRegistrationFormAddSection(RHManageRegFormBase): - """Add a section to the registration form""" + """Add a section to the registration form.""" def _process(self): section = RegistrationFormSection(registration_form=self.regform) @@ -47,7 +45,7 @@ def _process(self): class RHRegistrationFormModifySection(RHManageRegFormSectionBase): - """Delete/modify a section""" + """Delete/modify a section.""" def _process_DELETE(self): if self.section.type == RegistrationFormItemType.section_pd: @@ -59,9 +57,9 @@ def _process_DELETE(self): def _process_PATCH(self): changes = request.json['changes'] - if set(changes.viewkeys()) > {'title', 'description'}: + if set(changes.keys()) > {'title', 'description'}: raise BadRequest - for field, value in changes.iteritems(): + for field, value in changes.items(): setattr(self.section, field, value) db.session.flush() logger.info('Section %s modified by %s: %s', self.section, session.user, changes) @@ -69,7 +67,7 @@ def _process_PATCH(self): class RHRegistrationFormToggleSection(RHManageRegFormSectionBase): - """Enable/disable a section""" + """Enable/disable a section.""" def _process_POST(self): enabled = request.args.get('enable') == 'true' @@ -86,7 +84,7 @@ def _process_POST(self): class RHRegistrationFormMoveSection(RHManageRegFormSectionBase): - """Move a section within the registration form""" + """Move a section within the registration form.""" def _process(self): new_position = request.json['endPos'] + 1 @@ -103,8 +101,11 @@ def fn(section): return (old_position < section.position <= new_position and section.id != self.section.id and not section.is_deleted and section.is_enabled) start_enum = self.section.position - to_update = filter(fn, RegistrationFormSection.find(registration_form=self.regform, is_deleted=False) - .order_by(RegistrationFormSection.position).all()) + to_update = list(filter(fn, + RegistrationFormSection.query + .filter_by(registration_form=self.regform, is_deleted=False) + .order_by(RegistrationFormSection.position) + .all())) self.section.position = new_position for pos, section in enumerate(to_update, start_enum): section.position = pos diff --git a/indico/modules/events/registration/controllers/management/tickets.py b/indico/modules/events/registration/controllers/management/tickets.py index 2ef33f0fe6a..260b12a6f68 100644 --- a/indico/modules/events/registration/controllers/management/tickets.py +++ b/indico/modules/events/registration/controllers/management/tickets.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from io import BytesIO import qrcode @@ -14,11 +12,11 @@ from indico.core.config import config from indico.core.db import db +from indico.core.oauth.models.applications import OAuthApplication, SystemAppType from indico.modules.designer import PageOrientation, PageSize from indico.modules.events.registration.controllers.management import RHManageRegFormBase from indico.modules.events.registration.forms import TicketsForm -from indico.modules.oauth.models.applications import OAuthApplication, SystemAppType -from indico.web.flask.util import send_file, url_for +from indico.web.flask.util import send_file from indico.web.util import jsonify_data, jsonify_template @@ -61,17 +59,16 @@ def _process(self): border=1 ) - checkin_app = OAuthApplication.find_one(system_app_type=SystemAppType.checkin) + checkin_app = OAuthApplication.query.filter_by(system_app_type=SystemAppType.checkin).one() qr_data = { - "event_id": self.event.id, - "title": self.event.title, - "date": self.event.start_dt.isoformat(), - "version": 1, - "server": { - "base_url": config.BASE_URL, - "consumer_key": checkin_app.client_id, - "auth_url": url_for('oauth.oauth_authorize', _external=True), - "token_url": url_for('oauth.oauth_token', _external=True) + 'event_id': self.event.id, + 'title': self.event.title, + 'date': self.event.start_dt.isoformat(), + 'version': 2, + 'server': { + 'base_url': config.BASE_URL, + 'client_id': checkin_app.client_id, + 'scope': 'registrants', } } json_qr_data = json.dumps(qr_data) diff --git a/indico/modules/events/registration/fields/__init__.py b/indico/modules/events/registration/fields/__init__.py index 6cfaf21df0c..d8dbbc3c39f 100644 --- a/indico/modules/events/registration/fields/__init__.py +++ b/indico/modules/events/registration/fields/__init__.py @@ -1,18 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - - def get_field_types(): - """Get a dict with all registration field types""" - from .simple import (TextField, NumberField, TextAreaField, CheckboxField, DateField, BooleanField, PhoneField, - CountryField, FileField, EmailField) - from .choices import SingleChoiceField, AccommodationField, MultiChoiceField + """Get a dict with all registration field types.""" + from .choices import AccommodationField, MultiChoiceField, SingleChoiceField + from .simple import (BooleanField, CheckboxField, CountryField, DateField, EmailField, FileField, NumberField, + PhoneField, TextAreaField, TextField) return {field.name: field for field in (TextField, NumberField, TextAreaField, SingleChoiceField, CheckboxField, DateField, BooleanField, PhoneField, CountryField, FileField, EmailField, AccommodationField, MultiChoiceField)} diff --git a/indico/modules/events/registration/fields/base.py b/indico/modules/events/registration/fields/base.py index 436267ae510..2fcf8fd25d4 100644 --- a/indico/modules/events/registration/fields/base.py +++ b/indico/modules/events/registration/fields/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from copy import deepcopy from wtforms.validators import DataRequired, Optional @@ -14,8 +12,8 @@ from indico.modules.events.registration.models.registrations import RegistrationData -class RegistrationFormFieldBase(object): - """Base class for a registration form field definition""" +class RegistrationFormFieldBase: + """Base class for a registration form field definition.""" #: unique name of the field type name = None @@ -39,7 +37,7 @@ def default_value(self): @property def validators(self): - """Returns a list of validators for this field""" + """Return a list of validators for this field.""" return None @property @@ -47,7 +45,7 @@ def filter_choices(self): return None def calculate_price(self, reg_data, versioned_data): - """Calculates the price of the field + """Calculate the price of the field. :param reg_data: The user data for the field :param versioned_data: The versioned field data to use @@ -56,7 +54,7 @@ def calculate_price(self, reg_data, versioned_data): def create_sql_filter(self, data_list): """ - Creates a SQL criterion to check whether the field's value is + Create a SQL criterion to check whether the field's value is in `data_list`. The function is expected to return an operation on ``Registrationdata.data``. """ @@ -94,7 +92,7 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l @classmethod def process_field_data(cls, data, old_data=None, old_versioned_data=None): - """Processes the settings of the field. + """Process the settings of the field. :param data: The field data from the client :param old_data: The old unversioned field data (if available) @@ -105,8 +103,8 @@ def process_field_data(cls, data, old_data=None, old_versioned_data=None): data = dict(data) if 'places_limit' in data: data['places_limit'] = int(data['places_limit']) if data['places_limit'] else 0 - versioned_data = {k: v for k, v in data.iteritems() if k in cls.versioned_data_fields} - unversioned_data = {k: v for k, v in data.iteritems() if k not in cls.versioned_data_fields} + versioned_data = {k: v for k, v in data.items() if k in cls.versioned_data_fields} + unversioned_data = {k: v for k, v in data.items() if k not in cls.versioned_data_fields} return unversioned_data, versioned_data @classmethod @@ -118,7 +116,7 @@ def view_data(self): return self.unprocess_field_data(self.form_item.versioned_data, self.form_item.data) def get_friendly_data(self, registration_data, for_humans=False, for_search=False): - """Return the data contained in the field + """Return the data contained in the field. If for_humans is True, return a human-readable string representation. If for_search is True, return a string suitable for comparison in search. @@ -126,13 +124,13 @@ def get_friendly_data(self, registration_data, for_humans=False, for_search=Fals return registration_data.data def iter_placeholder_info(self): - yield None, 'Value of "{}" ({})'.format(self.form_item.title, self.form_item.parent.title) + yield None, f'Value of "{self.form_item.title}" ({self.form_item.parent.title})' def render_placeholder(self, data, key=None): return self.get_friendly_data(data) def get_places_used(self): - """Returns the number of used places for the field""" + """Return the number of used places for the field.""" return 0 @@ -142,7 +140,7 @@ def process_field_data(cls, data, old_data=None, old_versioned_data=None): data = deepcopy(data) data.setdefault('is_billable', False) data['price'] = float(data['price']) if data.get('price') else 0 - return super(RegistrationFormBillableField, cls).process_field_data(data, old_data, old_versioned_data) + return super().process_field_data(data, old_data, old_versioned_data) def calculate_price(self, reg_data, versioned_data): return versioned_data.get('price', 0) if versioned_data.get('is_billable') else 0 @@ -152,13 +150,13 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l new_data_version = self.form_item.current_data if billable_items_locked and old_data.price != self.calculate_price(value, new_data_version.versioned_data): return {} - return super(RegistrationFormBillableField, self).process_form_data(registration, value, old_data) + return super().process_form_data(registration, value, old_data) class RegistrationFormBillableItemsField(RegistrationFormBillableField): @classmethod def process_field_data(cls, data, old_data=None, old_versioned_data=None): - unversioned_data, versioned_data = super(RegistrationFormBillableItemsField, cls).process_field_data( + unversioned_data, versioned_data = super().process_field_data( data, old_data, old_versioned_data) # we don't have field-level billing data here del versioned_data['is_billable'] diff --git a/indico/modules/events/registration/fields/choices.py b/indico/modules/events/registration/fields/choices.py index 195ee30cbf8..2145d3096c3 100644 --- a/indico/modules/events/registration/fields/choices.py +++ b/indico/modules/events/registration/fields/choices.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import Counter from copy import deepcopy from datetime import date, datetime @@ -33,7 +31,7 @@ def get_field_merged_options(field, registration_data): result['modifiedChoice'] = [] if not rdata or not rdata.data: return result - values = [rdata.data['choice']] if 'choice' in rdata.data else rdata.data.keys() + values = [rdata.data['choice']] if 'choice' in rdata.data else list(rdata.data.keys()) for val in values: if val and not any(item['id'] == val for item in result['choices']): field_data = rdata.field_data @@ -79,7 +77,7 @@ def filter_choices(self): @property def view_data(self): - return dict(super(ChoiceBaseField, self).view_data, places_used=self.get_places_used()) + return dict(super().view_data, places_used=self.get_places_used()) @property def validators(self): @@ -109,15 +107,14 @@ def _check_number_of_places(form, field): @classmethod def process_field_data(cls, data, old_data=None, old_versioned_data=None): - unversioned_data, versioned_data = super(ChoiceBaseField, cls).process_field_data(data, old_data, - old_versioned_data) + unversioned_data, versioned_data = super().process_field_data(data, old_data, old_versioned_data) items = [x for x in versioned_data['choices'] if not x.get('remove')] captions = dict(old_data['captions']) if old_data is not None else {} if cls.has_default_item: unversioned_data.setdefault('default_item', None) for item in items: if 'id' not in item: - item['id'] = unicode(uuid4()) + item['id'] = str(uuid4()) item.setdefault('is_billable', False) item['price'] = float(item['price']) if item.get('price') else 0 item['places_limit'] = int(item['places_limit']) if item.get('places_limit') else 0 @@ -131,6 +128,8 @@ def process_field_data(cls, data, old_data=None, old_versioned_data=None): def get_places_used(self): places_used = Counter() + if not any(x.get('places_limit') for x in self.form_item.versioned_data['choices']): + return dict(places_used) for registration in self.form_item.registration_form.active_registrations: if self.form_item.id not in registration.data_by_field: continue @@ -173,9 +172,9 @@ def default_value(self): def get_friendly_data(self, registration_data, for_humans=False, for_search=False): if not registration_data.data: return '' - uuid, number_of_slots = registration_data.data.items()[0] + uuid, number_of_slots = list(registration_data.data.items())[0] caption = registration_data.field_data.field.data['captions'][uuid] - return '{} (+{})'.format(caption, number_of_slots - 1) if number_of_slots > 1 else caption + return f'{caption} (+{number_of_slots - 1})' if number_of_slots > 1 else caption def process_form_data(self, registration, value, old_data=None, billable_items_locked=False, new_data_version=None): if billable_items_locked and old_data.price: @@ -184,12 +183,11 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l # always store no-option as empty dict if value is None: value = {} - return super(SingleChoiceField, self).process_form_data(registration, value, old_data, billable_items_locked, - new_data_version) + return super().process_form_data(registration, value, old_data, billable_items_locked, new_data_version) def _hashable_choice(choice): - return frozenset(choice.iteritems()) + return frozenset(choice.items()) class MultiChoiceField(ChoiceBaseField): @@ -202,12 +200,12 @@ def default_value(self): def get_friendly_data(self, registration_data, for_humans=False, for_search=False): def _format_item(uuid, number_of_slots): caption = self.form_item.data['captions'][uuid] - return '{} (+{})'.format(caption, number_of_slots - 1) if number_of_slots > 1 else caption + return f'{caption} (+{number_of_slots - 1})' if number_of_slots > 1 else caption reg_data = registration_data.data if not reg_data: return '' - choices = sorted(_format_item(uuid, number_of_slots) for uuid, number_of_slots in reg_data.iteritems()) + choices = sorted(_format_item(uuid, number_of_slots) for uuid, number_of_slots in reg_data.items()) return ', '.join(choices) if for_humans or for_search else choices def process_form_data(self, registration, value, old_data=None, billable_items_locked=False, new_data_version=None): @@ -229,17 +227,17 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l selected_choice_hashes.update({c['id']: _hashable_choice(c) for c in self.form_item.versioned_data['choices'] if c['id'] in value and c['id'] not in selected_choice_hashes}) - selected_choice_hashes = set(selected_choice_hashes.itervalues()) + selected_choice_hashes = set(selected_choice_hashes.values()) existing_version_hashes = {c['id']: _hashable_choice(c) for c in old_data.field_data.versioned_data['choices']} latest_version_hashes = {c['id']: _hashable_choice(c) for c in self.form_item.versioned_data['choices']} - deselected_ids = old_data.data.viewkeys() - value.viewkeys() + deselected_ids = old_data.data.keys() - value.keys() modified_deselected = any(latest_version_hashes.get(id_) != existing_version_hashes.get(id_) for id_ in deselected_ids) - if selected_choice_hashes <= set(latest_version_hashes.itervalues()): + if selected_choice_hashes <= set(latest_version_hashes.values()): # all choices available in the latest version - upgrade to that version return_value['field_data'] = self.form_item.current_data - elif not modified_deselected and selected_choice_hashes <= set(existing_version_hashes.itervalues()): + elif not modified_deselected and selected_choice_hashes <= set(existing_version_hashes.values()): # all choices available in the previously selected version - stay with it return_value['field_data'] = old_data.field_data else: @@ -268,9 +266,9 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l new_choices = return_value['field_data'].versioned_data['choices'] if not billable_items_locked: - processed_data = super(MultiChoiceField, self).process_form_data(registration, value, old_data, False, - return_value.get('field_data')) - return {key: return_value.get(key, value) for key, value in processed_data.iteritems()} + processed_data = super().process_form_data(registration, value, old_data, False, + return_value.get('field_data')) + return {key: return_value.get(key, value) for key, value in processed_data.items()} # XXX: This code still relies on the client sending data for the disabled fields. # This is pretty ugly but especially in case of non-billable extra slots it makes # sense to keep it like this. If someone tampers with the list of billable fields @@ -278,9 +276,9 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l if has_old_data: old_choices_mapping = {x['id']: x for x in old_data.field_data.versioned_data['choices']} new_choices_mapping = {x['id']: x for x in new_choices} - old_billable = {uuid: num for uuid, num in old_data.data.iteritems() + old_billable = {uuid: num for uuid, num in old_data.data.items() if old_choices_mapping[uuid]['is_billable'] and old_choices_mapping[uuid]['price']} - new_billable = {uuid: num for uuid, num in value.iteritems() + new_billable = {uuid: num for uuid, num in value.items() if new_choices_mapping[uuid]['is_billable'] and new_choices_mapping[uuid]['price']} if has_old_data and old_billable != new_billable: # preserve existing data @@ -290,9 +288,9 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l # TODO: check item prices (in case there's a change between old/new version) # for now we simply ignore field changes in this case (since the old/new price # check in the base method will fail) - processed_data = super(MultiChoiceField, self).process_form_data(registration, value, old_data, True, - return_value.get('field_data')) - return {key: return_value.get(key, value) for key, value in processed_data.iteritems()} + processed_data = super().process_form_data(registration, value, old_data, True, + return_value.get('field_data')) + return {key: return_value.get(key, value) for key, value in processed_data.items()} def _to_machine_date(date): @@ -310,13 +308,12 @@ class AccommodationField(RegistrationFormBillableItemsField): @classmethod def process_field_data(cls, data, old_data=None, old_versioned_data=None): - unversioned_data, versioned_data = super(AccommodationField, cls).process_field_data(data, old_data, - old_versioned_data) + unversioned_data, versioned_data = super().process_field_data(data, old_data, old_versioned_data) items = [x for x in versioned_data['choices'] if not x.get('remove')] captions = dict(old_data['captions']) if old_data is not None else {} for item in items: if 'id' not in item: - item['id'] = unicode(uuid4()) + item['id'] = str(uuid4()) item.setdefault('is_billable', False) item['price'] = float(item['price']) if item.get('price') else 0 item['places_limit'] = int(item['places_limit']) if item.get('places_limit') else 0 @@ -377,7 +374,7 @@ def _check_number_of_places(form, field): @property def view_data(self): - return dict(super(AccommodationField, self).view_data, places_used=self.get_places_used()) + return dict(super().view_data, places_used=self.get_places_used()) def get_friendly_data(self, registration_data, for_humans=False, for_search=False): friendly_data = dict(registration_data.data) @@ -417,11 +414,12 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l if not is_no_accommodation: data.update({'arrival_date': value['arrivalDate'], 'departure_date': value['departureDate']}) - return super(AccommodationField, self).process_form_data(registration, data, old_data, billable_items_locked, - new_data_version) + return super().process_form_data(registration, data, old_data, billable_items_locked, new_data_version) def get_places_used(self): places_used = Counter() + if not any(x.get('places_limit') for x in self.form_item.versioned_data['choices']): + return dict(places_used) for registration in self.form_item.registration_form.active_registrations: if self.form_item.id not in registration.data_by_field: continue @@ -432,10 +430,10 @@ def get_places_used(self): return dict(places_used) def iter_placeholder_info(self): - yield 'name', 'Accommodation name for "{}" ({})'.format(self.form_item.title, self.form_item.parent.title) - yield 'nights', 'Number of nights for "{}" ({})'.format(self.form_item.title, self.form_item.parent.title) - yield 'arrival', 'Arrival date for "{}" ({})'.format(self.form_item.title, self.form_item.parent.title) - yield 'departure', 'Departure date for "{}" ({})'.format(self.form_item.title, self.form_item.parent.title) + yield 'name', f'Accommodation name for "{self.form_item.title}" ({self.form_item.parent.title})' + yield 'nights', f'Number of nights for "{self.form_item.title}" ({self.form_item.parent.title})' + yield 'arrival', f'Arrival date for "{self.form_item.title}" ({self.form_item.parent.title})' + yield 'departure', f'Departure date for "{self.form_item.title}" ({self.form_item.parent.title})' def render_placeholder(self, data, key=None): mapping = {'name': 'choice', @@ -443,6 +441,4 @@ def render_placeholder(self, data, key=None): 'arrival': 'arrival_date', 'departure': 'departure_date'} rv = self.get_friendly_data(data).get(mapping[key], '') - if isinstance(rv, date): - rv = format_date(rv).decode('utf-8') - return rv + return format_date(rv) if isinstance(rv, date) else rv diff --git a/indico/modules/events/registration/fields/choices_test.py b/indico/modules/events/registration/fields/choices_test.py index 80b3353fcae..ca2647bae73 100644 --- a/indico/modules/events/registration/fields/choices_test.py +++ b/indico/modules/events/registration/fields/choices_test.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from copy import deepcopy import pytest @@ -18,7 +16,7 @@ def _id(n): assert 0 <= n < 10 - return '{}0000000-0000-0000-0000-000000000000'.format(n) + return f'{n}0000000-0000-0000-0000-000000000000' @pytest.fixture @@ -37,7 +35,7 @@ def multi_choice_field(): def _update_data(data, changes): data = dict(deepcopy(data)) refs = {x['id']: x for x in data['choices']} - for id_, item_changes in changes.iteritems(): + for id_, item_changes in changes.items(): if id_ not in refs: entry = {'id': id_, 'places_limit': 0, 'is_billable': False, 'price': 0} entry.update(item_changes) diff --git a/indico/modules/events/registration/fields/simple.py b/indico/modules/events/registration/fields/simple.py index 33d487f9a0e..acc51692001 100644 --- a/indico/modules/events/registration/fields/simple.py +++ b/indico/modules/events/registration/fields/simple.py @@ -1,14 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import mimetypes -from collections import OrderedDict from datetime import datetime from operator import itemgetter @@ -84,11 +81,11 @@ def get_places_used(self): @property def view_data(self): - return dict(super(CheckboxField, self).view_data, places_used=self.get_places_used()) + return dict(super().view_data, places_used=self.get_places_used()) @property def filter_choices(self): - return {unicode(val).lower(): caption for val, caption in self.friendly_data_mapping.iteritems() + return {str(val).lower(): caption for val, caption in self.friendly_data_mapping.items() if val is not None} @property @@ -117,7 +114,7 @@ def process_form_data(self, registration, value, old_data=None, billable_items_l if value: date_format = self.form_item.data['date_format'] value = datetime.strptime(value, date_format).isoformat() - return super(DateField, self).process_form_data(registration, value, old_data, billable_items_locked) + return super().process_form_data(registration, value, old_data, billable_items_locked) def get_friendly_data(self, registration_data, for_humans=False, for_search=False): date_string = registration_data.data @@ -131,7 +128,7 @@ def get_friendly_data(self, registration_data, for_humans=False, for_search=Fals @property def view_data(self): has_time = ' ' in self.form_item.data['date_format'] - return dict(super(DateField, self).view_data, has_time=has_time) + return dict(super().view_data, has_time=has_time) class BooleanField(RegistrationFormBillableField): @@ -149,12 +146,12 @@ def wtf_field_kwargs(self): @property def filter_choices(self): - return {unicode(val).lower(): caption for val, caption in self.friendly_data_mapping.iteritems() + return {str(val).lower(): caption for val, caption in self.friendly_data_mapping.items() if val is not None} @property def view_data(self): - return dict(super(BooleanField, self).view_data, places_used=self.get_places_used()) + return dict(super().view_data, places_used=self.get_places_used()) @property def validators(self): @@ -204,17 +201,17 @@ class CountryField(RegistrationFormFieldBase): @property def wtf_field_kwargs(self): - return {'choices': sorted(get_countries().iteritems(), key=itemgetter(1))} + return {'choices': sorted(get_countries().items(), key=itemgetter(1))} @classmethod def unprocess_field_data(cls, versioned_data, unversioned_data): - choices = sorted(({'caption': v, 'countryKey': k} for k, v in get_countries().iteritems()), + choices = sorted(({'caption': v, 'countryKey': k} for k, v in get_countries().items()), key=itemgetter('caption')) return {'choices': choices} @property def filter_choices(self): - return OrderedDict(self.wtf_field_kwargs['choices']) + return dict(self.wtf_field_kwargs['choices']) def get_friendly_data(self, registration_data, for_humans=False, for_search=False): if registration_data.data == 'None': diff --git a/indico/modules/events/registration/forms.py b/indico/modules/events/registration/forms.py index f579ec609d2..abaceb162ae 100644 --- a/indico/modules/events/registration/forms.py +++ b/indico/modules/events/registration/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import time from operator import itemgetter @@ -19,6 +17,7 @@ from indico.core import signals from indico.core.config import config +from indico.core.db import db from indico.modules.designer import PageLayout, PageOrientation, PageSize, TemplateType from indico.modules.designer.util import get_default_ticket_on_category, get_inherited_templates from indico.modules.events.features.util import is_feature_enabled @@ -45,8 +44,8 @@ def _check_if_payment_required(form, field): class RegistrationFormForm(IndicoForm): _price_fields = ('currency', 'base_price') - _registrant_notification_fields = ('notification_sender_address', - 'message_pending', 'message_unpaid', 'message_complete') + _registrant_notification_fields = ('notification_sender_address', 'message_pending', 'message_unpaid', + 'message_complete', 'attach_ical') _manager_notification_fields = ('manager_notifications_enabled', 'manager_notification_recipients') _special_fields = _price_fields + _registrant_notification_fields + _manager_notification_fields @@ -83,12 +82,23 @@ class RegistrationFormForm(IndicoForm): currency = SelectField(_('Currency'), [DataRequired()], description=_('The currency for new registrations')) notification_sender_address = StringField(_('Notification sender address'), [IndicoEmail()], filters=[lambda x: (x or None)]) - message_pending = TextAreaField(_("Message for pending registrations"), - description=_("Text included in emails sent to pending registrations")) - message_unpaid = TextAreaField(_("Message for unpaid registrations"), - description=_("Text included in emails sent to unpaid registrations")) - message_complete = TextAreaField(_("Message for complete registrations"), - description=_("Text included in emails sent to complete registrations")) + message_pending = TextAreaField( + _("Message for pending registrations"), + description=_("Text included in emails sent to pending registrations (Markdown syntax)") + ) + message_unpaid = TextAreaField( + _("Message for unpaid registrations"), + description=_("Text included in emails sent to unpaid registrations (Markdown syntax)") + ) + message_complete = TextAreaField( + _("Message for complete registrations"), + description=_("Text included in emails sent to complete registrations (Markdown syntax)") + ) + attach_ical = BooleanField( + _('Attach iCalendar file'), + widget=SwitchWidget(), + description=_('Attach an iCalendar file to the mail sent once a registration is complete') + ) manager_notifications_enabled = BooleanField(_('Enabled'), widget=SwitchWidget(), description=_("Enable notifications to managers about registrations")) manager_notification_recipients = EmailListField(_('List of recipients'), @@ -98,11 +108,11 @@ class RegistrationFormForm(IndicoForm): def __init__(self, *args, **kwargs): self.event = kwargs.pop('event') - super(RegistrationFormForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self._set_currencies() self.notification_sender_address.description = _('Email address set as the sender of all ' 'notifications sent to users. If empty, ' - 'then {0} is used.'.format(config.NO_REPLY_EMAIL)) + 'then {email} is used.').format(email=config.NO_REPLY_EMAIL) def _set_currencies(self): currencies = [(c['code'], '{0[code]} ({0[name]})'.format(c)) for c in payment_settings.get('currencies')] @@ -122,7 +132,7 @@ class RegistrationFormScheduleForm(IndicoForm): def __init__(self, *args, **kwargs): regform = kwargs.pop('regform') self.timezone = regform.event.timezone - super(RegistrationFormScheduleForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) class InvitationFormBase(IndicoForm): @@ -137,10 +147,10 @@ class InvitationFormBase(IndicoForm): def __init__(self, *args, **kwargs): self.regform = kwargs.pop('regform') event = self.regform.event - super(InvitationFormBase, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) if not self.regform.moderation_enabled: del self.skip_moderation - self.email_from.choices = event.get_allowed_sender_emails().items() + self.email_from.choices = list(event.get_allowed_sender_emails().items()) self.email_body.description = render_placeholder_info('registration-invitation-email', invitation=None) def validate_email_body(self, field): @@ -168,9 +178,9 @@ def users(self): 'affiliation': self.affiliation.data}] def validate_email(self, field): - if RegistrationInvitation.find(email=field.data).with_parent(self.regform).count(): + if RegistrationInvitation.query.filter_by(email=field.data).with_parent(self.regform).has_rows(): raise ValidationError(_("There is already an invitation with this email address.")) - if Registration.find(email=field.data, is_active=True).with_parent(self.regform).count(): + if Registration.query.filter_by(email=field.data, is_active=True).with_parent(self.regform).has_rows(): raise ValidationError(_("There is already a registration with this email address.")) @@ -219,8 +229,8 @@ class EmailRegistrantsForm(IndicoForm): def __init__(self, *args, **kwargs): self.regform = kwargs.pop('regform') event = self.regform.event - super(EmailRegistrantsForm, self).__init__(*args, **kwargs) - self.from_address.choices = event.get_allowed_sender_emails().items() + super().__init__(*args, **kwargs) + self.from_address.choices = list(event.get_allowed_sender_emails().items()) self.body.description = render_placeholder_info('registration-email', regform=self.regform, registration=None) def validate_body(self, field): @@ -229,7 +239,7 @@ def validate_body(self, field): raise ValidationError(_('Missing placeholders: {}').format(', '.join(missing))) def is_submitted(self): - return super(EmailRegistrantsForm, self).is_submitted() and 'submitted' in request.form + return super().is_submitted() and 'submitted' in request.form class TicketsForm(IndicoForm): @@ -256,7 +266,7 @@ class TicketsForm(IndicoForm): def __init__(self, *args, **kwargs): event = kwargs.pop('event') - super(TicketsForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) default_tpl = get_default_ticket_on_category(event.category) all_templates = set(event.designer_templates) | get_inherited_templates(event) badge_templates = [(tpl.id, tpl.title) for tpl in all_templates @@ -288,11 +298,14 @@ def validate_json(self, field): try: jsonschema.validate(field.data, schema) except jsonschema.ValidationError as exc: - raise ValidationError(exc.message) + raise ValidationError(str(exc)) class ParticipantsDisplayFormColumnsForm(IndicoForm): - """Form to customize the columns for a particular registration form on the participant list.""" + """ + Form to customize the columns for a particular registration form + on the participant list. + """ json = JSONField() def validate_json(self, field): @@ -308,11 +321,13 @@ def validate_json(self, field): try: jsonschema.validate(field.data, schema) except jsonschema.ValidationError as exc: - raise ValidationError(exc.message) + raise ValidationError(str(exc)) class RegistrationManagersForm(IndicoForm): - """Form to manage users with privileges to modify registration-related items""" + """ + Form to manage users with privileges to modify registration-related items. + """ managers = PrincipalListField(_('Registration managers'), allow_groups=True, allow_emails=True, allow_external_users=True, @@ -321,13 +336,15 @@ class RegistrationManagersForm(IndicoForm): def __init__(self, *args, **kwargs): self.event = kwargs.pop('event') - super(RegistrationManagersForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) class CreateMultipleRegistrationsForm(IndicoForm): - """Form to create multiple registrations of Indico users at the same time.""" + """ + Form to create multiple registrations of Indico users at the same time. + """ - user_principals = PrincipalListField(_("Indico users"), [DataRequired()]) + user_principals = PrincipalListField(_("Indico users"), [DataRequired()], allow_external_users=True) notify_users = BooleanField(_("Send e-mail notifications"), default=True, description=_("Notify the users about the registration."), @@ -336,13 +353,15 @@ class CreateMultipleRegistrationsForm(IndicoForm): def __init__(self, *args, **kwargs): self._regform = kwargs.pop('regform') open_add_user_dialog = kwargs.pop('open_add_user_dialog', False) - super(CreateMultipleRegistrationsForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) self.user_principals.open_immediately = open_add_user_dialog def validate_user_principals(self, field): for user in field.data: - if user.registrations.filter_by(registration_form=self._regform, is_deleted=False).one_or_none(): + if user in db.session and self._regform.get_registration(user=user): raise ValidationError(_("A registration for {} already exists.").format(user.full_name)) + elif self._regform.get_registration(email=user.email): + raise ValidationError(_("A registration for {} already exists.").format(user.email)) class BadgeSettingsForm(IndicoForm): @@ -372,14 +391,14 @@ def __init__(self, event, **kwargs): badge_templates = [tpl for tpl in all_templates if tpl.type.name == 'badge'] signals.event.filter_selectable_badges.send(type(self), badge_templates=badge_templates) tickets = kwargs.pop('tickets') - super(BadgeSettingsForm, self).__init__(**kwargs) - self.template.choices = sorted(((unicode(tpl.id), tpl.title) + super().__init__(**kwargs) + self.template.choices = sorted(((str(tpl.id), tpl.title) for tpl in badge_templates if tpl.is_ticket == tickets), key=itemgetter(1)) def is_submitted(self): - return super(BadgeSettingsForm, self).is_submitted() and 'submitted' in request.form + return super().is_submitted() and 'submitted' in request.form class ImportRegistrationsForm(IndicoForm): @@ -391,6 +410,16 @@ class ImportRegistrationsForm(IndicoForm): def __init__(self, *args, **kwargs): self.regform = kwargs.pop('regform') - super(ImportRegistrationsForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) if not self.regform.moderation_enabled: del self.skip_moderation + + +class RejectRegistrantsForm(IndicoForm): + rejection_reason = TextAreaField(_('Reason'), description=_("You can provide a reason for the rejection here.")) + attach_rejection_reason = BooleanField(_('Attach reason'), widget=SwitchWidget()) + registration_id = HiddenFieldList() + submitted = HiddenField() + + def is_submitted(self): + return super().is_submitted() and 'submitted' in request.form diff --git a/indico/modules/events/registration/lists.py b/indico/modules/events/registration/lists.py index 3a3ed1ef8b8..c262adad639 100644 --- a/indico/modules/events/registration/lists.py +++ b/indico/modules/events/registration/lists.py @@ -1,14 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - -from collections import OrderedDict - from flask import request from sqlalchemy.orm import joinedload @@ -28,37 +24,37 @@ class RegistrationListGenerator(ListGeneratorBase): list_link_type = 'registration' def __init__(self, regform): - super(RegistrationListGenerator, self).__init__(regform.event, entry_parent=regform) + super().__init__(regform.event, entry_parent=regform) self.regform = regform self.default_list_config = { 'items': ('title', 'email', 'affiliation', 'reg_date', 'state'), 'filters': {'fields': {}, 'items': {}} } - self.static_items = OrderedDict([ - ('reg_date', { + self.static_items = { + 'reg_date': { 'title': _('Registration Date'), - }), - ('price', { + }, + 'price': { 'title': _('Price'), - }), - ('state', { + }, + 'state': { 'title': _('State'), 'filter_choices': {str(state.value): state.title for state in RegistrationState} - }), - ('checked_in', { + }, + 'checked_in': { 'title': _('Checked in'), 'filter_choices': { '0': _('No'), '1': _('Yes') } - }), - ('checked_in_date', { + }, + 'checked_in_date': { 'title': _('Check-in date'), - }), - ('payment_date', { + }, + 'payment_date': { 'title': _('Payment date'), - }) - ]) + }, + } self.personal_items = ('title', 'first_name', 'last_name', 'email', 'position', 'affiliation', 'address', 'phone', 'country') self.list_config = self._get_config() @@ -73,8 +69,8 @@ def _get_static_columns(self, ids): result = [] for item_id in ids: if item_id in self.personal_items: - field = RegistrationFormItem.find_one(registration_form=self.regform, - personal_data_type=PersonalDataType[item_id]) + field = RegistrationFormItem.query.filter_by(registration_form=self.regform, + personal_data_type=PersonalDataType[item_id]).one() result.append({ 'id': field.id, 'caption': field.title @@ -91,7 +87,7 @@ def _column_ids_to_db(self, ids): result = [] personal_data_field_ids = {x.personal_data_type: x.id for x in self.regform.form_items if x.is_field} for item_id in ids: - if isinstance(item_id, basestring): + if isinstance(item_id, str): personal_data_type = PersonalDataType.get(item_id) if personal_data_type: item_id = personal_data_field_ids[personal_data_type] @@ -99,12 +95,14 @@ def _column_ids_to_db(self, ids): return result def _get_sorted_regform_items(self, item_ids): - """Return the form items ordered by their position in the registration form.""" + """ + Return the form items ordered by their position in the registration form. + """ if not item_ids: return [] - return (RegistrationFormItem - .find(~RegistrationFormItem.is_deleted, RegistrationFormItem.id.in_(item_ids)) + return (RegistrationFormItem.query + .filter(~RegistrationFormItem.is_deleted, RegistrationFormItem.id.in_(item_ids)) .with_parent(self.regform) .join(RegistrationFormItem.parent, aliased=True) .filter(~RegistrationFormItem.is_deleted) # parent deleted @@ -114,10 +112,10 @@ def _get_sorted_regform_items(self, item_ids): .all()) def _get_filters_from_request(self): - filters = super(RegistrationListGenerator, self)._get_filters_from_request() + filters = super()._get_filters_from_request() for field in self.regform.form_items: if field.is_field and field.input_type in {'single_choice', 'multi_choice', 'country', 'bool', 'checkbox'}: - options = request.form.getlist('field_{}'.format(field.id)) + options = request.form.getlist(f'field_{field.id}') if options: filters['fields'][str(field.id)] = options return filters @@ -135,13 +133,13 @@ def _filter_list_entries(self, query, filters): field_types = {str(f.id): f.field_impl for f in self.regform.form_items if f.is_field and not f.is_deleted and (f.parent_id is None or not f.parent.is_deleted)} field_filters = {field_id: data_list - for field_id, data_list in filters['fields'].iteritems() + for field_id, data_list in filters['fields'].items() if field_id in field_types} if not field_filters and not filters['items']: return query criteria = [db.and_(RegistrationFormFieldData.field_id == field_id, field_types[field_id].create_sql_filter(data_list)) - for field_id, data_list in field_filters.iteritems()] + for field_id, data_list in field_filters.items()] items_criteria = [] if 'checked_in' in filters['items']: checked_in_values = filters['items']['checked_in'] @@ -160,7 +158,7 @@ def _filter_list_entries(self, query, filters): .filter(RegistrationData.registration_id == Registration.id) .filter(db.or_(*criteria)) .correlate(Registration) - .as_scalar()) + .scalar_subquery()) query = query.filter(subquery == len(field_filters)) return query.filter(db.or_(*items_criteria)) diff --git a/indico/modules/events/registration/logging.py b/indico/modules/events/registration/logging.py index 9e454493a00..60eddff4fd2 100644 --- a/indico/modules/events/registration/logging.py +++ b/indico/modules/events/registration/logging.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals @@ -23,7 +21,7 @@ def connect_log_signals(): def log_registration_check_in(registration, **kwargs): """Log a registration check-in action to the event log.""" if registration.checked_in: - log_text = '"{}" has been checked in'.format(registration.full_name) + log_text = f'"{registration.full_name}" has been checked in' else: log_text = '"{}" check-in has been reset' registration.log(EventLogRealm.participants, EventLogKind.change, 'Registration', @@ -35,6 +33,7 @@ def log_registration_updated(registration, previous_state, **kwargs): if not previous_state: return previous_state_title = orig_string(previous_state.title) + data = {'Previous state': previous_state_title} if (previous_state == RegistrationState.pending and registration.state in (RegistrationState.complete, RegistrationState.unpaid)): log_text = 'Registration for "{}" has been approved' @@ -42,6 +41,8 @@ def log_registration_updated(registration, previous_state, **kwargs): elif previous_state == RegistrationState.pending and registration.state == RegistrationState.rejected: log_text = 'Registration for "{}" has been rejected' kind = EventLogKind.negative + if registration.rejection_reason: + data['Reason'] = registration.rejection_reason elif previous_state == RegistrationState.unpaid and registration.state == RegistrationState.complete: log_text = 'Registration for "{}" has been paid' kind = EventLogKind.positive @@ -57,4 +58,4 @@ def log_registration_updated(registration, previous_state, **kwargs): state_title) kind = EventLogKind.change registration.log(EventLogRealm.participants, kind, 'Registration', log_text.format(registration.full_name), - session.user, data={'Previous state': previous_state_title}) + session.user, data=data) diff --git a/indico/modules/events/registration/models/form_fields.py b/indico/modules/events/registration/models/form_fields.py index aadf0b71677..c5790fd7e26 100644 --- a/indico/modules/events/registration/models/form_fields.py +++ b/indico/modules/events/registration/models/form_fields.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.event import listens_for from werkzeug.datastructures import ImmutableDict @@ -14,11 +12,11 @@ from indico.core.db import db from indico.modules.events.registration.fields import get_field_types from indico.modules.events.registration.models.items import RegistrationFormItem, RegistrationFormItemType -from indico.util.string import camelize_keys, return_ascii +from indico.util.string import camelize_keys class RegistrationFormFieldData(db.Model): - """Description of a registration form field""" + """Description of a registration form field.""" __tablename__ = 'form_field_data' __table_args__ = {'schema': 'event_registration'} @@ -45,13 +43,12 @@ class RegistrationFormFieldData(db.Model): # - field (RegistrationFormItem.data_versions) # - registration_data (RegistrationData.field_data) - @return_ascii def __repr__(self): - return ''.format(self.id, self.field_id) + return f'' class RegistrationFormField(RegistrationFormItem): - """A registration form field""" + """A registration form field.""" __mapper_args__ = { 'polymorphic_identity': RegistrationFormItemType.field @@ -88,13 +85,13 @@ def view_data(self): base_dict = dict(self.versioned_data, **self.data) base_dict.update(is_enabled=self.is_enabled, title=self.title, is_required=self.is_required, input_type=self.input_type, html_name=self.html_field_name, - **super(RegistrationFormField, self).view_data) + **super().view_data) base_dict.update(self.field_impl.view_data) return camelize_keys(base_dict) @property def html_field_name(self): - return 'field_{}'.format(self.id) + return f'field_{self.id}' def get_friendly_data(self, registration_data, **kwargs): return self.field_impl.get_friendly_data(registration_data, **kwargs) @@ -110,7 +107,7 @@ class RegistrationFormPersonalDataField(RegistrationFormField): @property def view_data(self): - data = dict(super(RegistrationFormPersonalDataField, self).view_data, + data = dict(super().view_data, field_is_required=self.personal_data_type.is_required, field_is_personal_data=True) return camelize_keys(data) diff --git a/indico/modules/events/registration/models/forms.py b/indico/modules/events/registration/models/forms.py index b2a5e38abfd..1c1eaf14507 100644 --- a/indico/modules/events/registration/models/forms.py +++ b/indico/modules/events/registration/models/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from uuid import UUID from babel.numbers import format_currency @@ -26,30 +24,24 @@ from indico.modules.events.registration.models.registrations import Registration, RegistrationState from indico.util.caching import memoize_request from indico.util.date_time import now_utc +from indico.util.enum import RichIntEnum from indico.util.i18n import L_ -from indico.util.string import return_ascii -from indico.util.struct.enum import RichIntEnum class ModificationMode(RichIntEnum): - __titles__ = [None, L_('Until modification deadline'), L_('Until payment'), L_('Never')] + __titles__ = [None, L_('Until modification deadline'), L_('Until payment'), L_('Never'), L_('Until approved')] allowed_always = 1 allowed_until_payment = 2 not_allowed = 3 + allowed_until_approved = 4 class RegistrationForm(db.Model): - """A registration form for an event""" + """A registration form for an event.""" __tablename__ = 'forms' principal_type = PrincipalType.registration_form principal_order = 2 - is_group = False - is_network = False - is_single_person = False - is_event_role = False - is_category_role = False - is_registration_form = True __table_args__ = (db.Index('ix_uq_forms_participation', 'event_id', unique=True, postgresql_where=db.text('is_participation AND NOT is_deleted')), @@ -193,6 +185,12 @@ class RegistrationForm(db.Model): nullable=False, default='' ) + #: If the completed registration email should include the event's iCalendar file. + attach_ical = db.Column( + db.Boolean, + nullable=False, + default=False + ) #: Whether the manager notifications for this event are enabled manager_notifications_enabled = db.Column( db.Boolean, @@ -309,9 +307,14 @@ def __contains__(self, user): ~RegistrationForm.is_deleted) .has_rows()) + @property + def name(self): + # needed when sorting acl entries by name + return self.title + @property def identifier(self): - return 'RegistrationForm:{}'.format(self.id) + return f'RegistrationForm:{self.id}' @hybrid_property def has_ended(self): @@ -396,16 +399,17 @@ def sender_address(self): contact_email = self.event.contact_emails[0] if self.event.contact_emails else None return self.notification_sender_address or contact_email - @return_ascii def __repr__(self): - return ''.format(self.id, self.event_id, self.title) + return f'' def is_modification_allowed(self, registration): - """Checks whether a registration may be modified""" + """Check whether a registration may be modified.""" if not registration.is_active: return False elif self.modification_mode == ModificationMode.allowed_always: return True + elif self.modification_mode == ModificationMode.allowed_until_approved: + return registration.state == RegistrationState.pending elif self.modification_mode == ModificationMode.allowed_until_payment: return not registration.is_paid else: @@ -416,7 +420,7 @@ def can_submit(self, user): @memoize_request def get_registration(self, user=None, uuid=None, email=None): - """Retrieves registrations for this registration form by user or uuid""" + """Retrieve registrations for this registration form by user or uuid.""" if (bool(user) + bool(uuid) + bool(email)) != 1: raise ValueError("Exactly one of `user`, `uuid` and `email` must be specified") if user: @@ -434,7 +438,7 @@ def render_base_price(self): return format_currency(self.base_price, self.currency, locale=session.lang or 'en_GB') def get_personal_data_field_id(self, personal_data_type): - """Returns the field id corresponding to the personal data field with the given name.""" + """Return the field id corresponding to the personal data field with the given name.""" for field in self.active_fields: if (isinstance(field, RegistrationFormPersonalDataField) and field.personal_data_type == personal_data_type): @@ -445,10 +449,12 @@ def get_personal_data_field_id(self, personal_data_type): def _mappers_configured(): query = (select([db.func.count(Registration.id)]) .where((Registration.registration_form_id == RegistrationForm.id) & Registration.is_active) - .correlate_except(Registration)) + .correlate_except(Registration) + .scalar_subquery()) RegistrationForm.active_registration_count = column_property(query, deferred=True) query = (select([db.func.count(Registration.id)]) .where((Registration.registration_form_id == RegistrationForm.id) & ~Registration.is_deleted) - .correlate_except(Registration)) + .correlate_except(Registration) + .scalar_subquery()) RegistrationForm.existing_registrations_count = column_property(query, deferred=True) diff --git a/indico/modules/events/registration/models/invitations.py b/indico/modules/events/registration/models/invitations.py index 883fb6306c9..57ed5180328 100644 --- a/indico/modules/events/registration/models/invitations.py +++ b/indico/modules/events/registration/models/invitations.py @@ -1,22 +1,20 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from uuid import uuid4 from sqlalchemy.dialects.postgresql import UUID from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum +from indico.util.enum import RichIntEnum from indico.util.i18n import L_ from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_repr class InvitationState(RichIntEnum): @@ -27,7 +25,7 @@ class InvitationState(RichIntEnum): class RegistrationInvitation(db.Model): - """An invitation for someone to register""" + """An invitation for someone to register.""" __tablename__ = 'invitations' __table_args__ = (db.CheckConstraint("(state = {state}) OR (registration_id IS NULL)" .format(state=InvitationState.accepted), name='registration_state'), @@ -45,7 +43,7 @@ class RegistrationInvitation(db.Model): index=True, unique=True, nullable=False, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) #: The ID of the registration form registration_form_id = db.Column( @@ -117,12 +115,11 @@ def locator(self): def locator(self): """A locator suitable for 'display' pages. - Instead of the numeric ID it uses the UUID + Instead of the numeric ID it uses the UUID. """ assert self.uuid is not None return dict(self.registration_form.locator, invitation=self.uuid) - @return_ascii def __repr__(self): - full_name = '{} {}'.format(self.first_name, self.last_name) + full_name = f'{self.first_name} {self.last_name}' return format_repr(self, 'id', 'registration_form_id', 'email', 'state', _text=full_name) diff --git a/indico/modules/events/registration/models/items.py b/indico/modules/events/registration/models/items.py index 9e02f170d3d..3c26e590d8d 100644 --- a/indico/modules/events/registration/models/items.py +++ b/indico/modules/events/registration/models/items.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from uuid import uuid4 from sqlalchemy import literal @@ -18,9 +16,9 @@ from indico.core.db.sqlalchemy import PyIntEnum from indico.modules.users.models.users import UserTitle from indico.util.decorators import strict_classproperty +from indico.util.enum import IndicoEnum from indico.util.i18n import orig_string -from indico.util.string import camelize_keys, format_repr, return_ascii -from indico.util.struct.enum import IndicoEnum +from indico.util.string import camelize_keys, format_repr def _get_next_position(context): @@ -28,7 +26,10 @@ def _get_next_position(context): regform_id = context.current_parameters['registration_form_id'] parent_id = context.current_parameters['parent_id'] res = (db.session.query(db.func.max(RegistrationFormItem.position)) - .filter_by(parent_id=parent_id, registration_form_id=regform_id, is_deleted=False, is_enabled=True) + .filter(RegistrationFormItem.parent_id == parent_id, + RegistrationFormItem.registration_form_id == regform_id, + ~RegistrationFormItem.is_deleted, + RegistrationFormItem.is_enabled) .one()) return (res[0] or 0) + 1 @@ -43,7 +44,9 @@ class RegistrationFormItemType(int, IndicoEnum): # We are not using a RichIntEnum since one of the instances is named "title". class PersonalDataType(int, IndicoEnum): - """Description of the personal data items that exist on every registration form""" + """ + Description of the personal data items that exist on every registration form. + """ __titles__ = [None, 'Email Address', 'First Name', 'Last Name', 'Affiliation', 'Title', 'Address', 'Phone Number', 'Country', 'Position'] @@ -68,16 +71,6 @@ def FIELD_DATA(cls): 'places_limit': 0, 'is_enabled': True} return [ - (cls.title, { - 'title': cls.title.get_title(), - 'input_type': 'single_choice', - 'data': { - 'item_type': 'dropdown', - 'with_extra_slots': False, - 'choices': [dict(title_item, id=unicode(uuid4()), caption=orig_string(t.title)) - for t in UserTitle if t] - } - }), (cls.first_name, { 'title': cls.first_name.get_title(), 'input_type': 'text' @@ -119,6 +112,18 @@ def FIELD_DATA(cls): 'is_enabled': False, 'position': 1003 }), + (cls.title, { + 'title': cls.title.get_title(), + 'input_type': 'single_choice', + 'is_enabled': False, + 'position': 1004, + 'data': { + 'item_type': 'dropdown', + 'with_extra_slots': False, + 'choices': [dict(title_item, id=str(uuid4()), caption=orig_string(t.title)) + for t in UserTitle if t] + } + }), ] @property @@ -138,14 +143,14 @@ def column(self): class RegistrationFormItem(db.Model): - """Generic registration form item""" + """Generic registration form item.""" __tablename__ = 'form_items' __table_args__ = ( db.CheckConstraint("(input_type IS NULL) = (type NOT IN ({t.field}, {t.field_pd}))" .format(t=RegistrationFormItemType), name='valid_input'), - db.CheckConstraint("NOT is_manager_only OR type = {type}".format(type=RegistrationFormItemType.section), + db.CheckConstraint(f"NOT is_manager_only OR type = {RegistrationFormItemType.section}", name='valid_manager_only'), db.CheckConstraint("(type IN ({t.section}, {t.section_pd})) = (parent_id IS NULL)" .format(t=RegistrationFormItemType), @@ -156,7 +161,7 @@ class RegistrationFormItem(db.Model): db.CheckConstraint("NOT is_deleted OR (type NOT IN ({t.section_pd}, {t.field_pd}))" .format(t=RegistrationFormItemType), name='pd_not_deleted'), - db.CheckConstraint("is_enabled OR type != {type}".format(type=RegistrationFormItemType.section_pd), + db.CheckConstraint(f"is_enabled OR type != {RegistrationFormItemType.section_pd}", name='pd_section_enabled'), db.CheckConstraint("is_enabled OR type != {type} OR personal_data_type NOT IN " "({pt.email}, {pt.first_name}, {pt.last_name})" @@ -170,9 +175,9 @@ class RegistrationFormItem(db.Model): .format(t=RegistrationFormItemType), name='current_data_id_only_field'), db.Index('ix_uq_form_items_pd_section', 'registration_form_id', unique=True, - postgresql_where=db.text('type = {type}'.format(type=RegistrationFormItemType.section_pd))), + postgresql_where=db.text(f'type = {RegistrationFormItemType.section_pd}')), db.Index('ix_uq_form_items_pd_field', 'registration_form_id', 'personal_data_type', unique=True, - postgresql_where=db.text('type = {type}'.format(type=RegistrationFormItemType.field_pd))), + postgresql_where=db.text(f'type = {RegistrationFormItemType.field_pd}')), {'schema': 'event_registration'} ) __mapper_args__ = { @@ -309,7 +314,7 @@ class RegistrationFormItem(db.Model): @property def view_data(self): - """Returns object with data that Angular can understand""" + """Return object with data that Angular can understand.""" return dict(id=self.id, description=self.description, position=self.position) @hybrid_property @@ -342,14 +347,13 @@ def is_visible(cls): .exists()) return cls.is_enabled & ~cls.is_deleted & ((cls.parent_id == None) | query) # noqa - @return_ascii def __repr__(self): return format_repr(self, 'id', 'registration_form_id', is_enabled=True, is_deleted=False, is_manager_only=False, _text=self.title) class RegistrationFormSection(RegistrationFormItem): - """Registration form section that can contain fields and text""" + """Registration form section that can contain fields and text.""" __mapper_args__ = { 'polymorphic_identity': RegistrationFormItemType.section @@ -369,7 +373,7 @@ def locator(self): @property def own_data(self): - field_data = dict(super(RegistrationFormSection, self).view_data, + field_data = dict(super().view_data, enabled=self.is_enabled, title=self.title, is_manager_only=self.is_manager_only, @@ -390,13 +394,13 @@ class RegistrationFormPersonalDataSection(RegistrationFormSection): @property def view_data(self): - field_data = dict(super(RegistrationFormPersonalDataSection, self).view_data, is_personal_data=True) + field_data = dict(super().view_data, is_personal_data=True) del field_data['isPersonalData'] return camelize_keys(field_data) class RegistrationFormText(RegistrationFormItem): - """Text to be displayed in registration form sections""" + """Text to be displayed in registration form sections.""" __mapper_args__ = { 'polymorphic_identity': RegistrationFormItemType.text @@ -408,6 +412,6 @@ def locator(self): @property def view_data(self): - field_data = dict(super(RegistrationFormText, self).view_data, is_enabled=self.is_enabled, input_type='label', + field_data = dict(super().view_data, is_enabled=self.is_enabled, input_type='label', title=self.title) return camelize_keys(field_data) diff --git a/indico/modules/events/registration/models/legacy_mapping.py b/indico/modules/events/registration/models/legacy_mapping.py index 94ea7718a15..25351857425 100644 --- a/indico/modules/events/registration/models/legacy_mapping.py +++ b/indico/modules/events/registration/models/legacy_mapping.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class LegacyRegistrationMapping(db.Model): - """Legacy registration id/token mapping + """Legacy registration id/token mapping. Legacy registrations had tokens which are not compatible with the new UUID-based ones. @@ -54,6 +52,5 @@ class LegacyRegistrationMapping(db.Model): ) ) - @return_ascii def __repr__(self): return format_repr(self, 'event_id', 'legacy_registrant_id', 'legacy_registrant_key', 'registration_id') diff --git a/indico/modules/events/registration/models/registrations.py b/indico/modules/events/registration/models/registrations.py index b3ebd78dd28..c20d6be7ea0 100644 --- a/indico/modules/events/registration/models/registrations.py +++ b/indico/modules/events/registration/models/registrations.py @@ -1,15 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import posixpath import time -from collections import OrderedDict from decimal import Decimal from uuid import uuid4 @@ -30,12 +27,13 @@ from indico.modules.users.models.users import format_display_full_name from indico.util.date_time import now_utc from indico.util.decorators import classproperty +from indico.util.enum import RichIntEnum from indico.util.fs import secure_filename from indico.util.i18n import L_ from indico.util.locators import locator_property from indico.util.signals import values_from_signal -from indico.util.string import format_full_name, format_repr, return_ascii, strict_unicode -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_full_name, format_repr, strict_str +from indico.web.flask.util import url_for class RegistrationState(RichIntEnum): @@ -56,7 +54,7 @@ def _get_next_friendly_id(context): class Registration(db.Model): - """Somebody's registration for an event through a registration form""" + """Somebody's registration for an event through a registration form.""" __tablename__ = 'registrations' __table_args__ = (db.CheckConstraint('email = lower(email)', 'lowercase_email'), db.Index(None, 'friendly_id', 'event_id', unique=True, @@ -80,7 +78,7 @@ class Registration(db.Model): index=True, unique=True, nullable=False, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) #: The human-friendly ID for the object friendly_id = db.Column( @@ -172,7 +170,7 @@ class Registration(db.Model): index=True, unique=True, nullable=False, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) #: Whether the person has checked in. Setting this also sets or clears #: `checked_in_dt`. @@ -186,7 +184,12 @@ class Registration(db.Model): UTCDateTime, nullable=True ) - + #: If given a reason for rejection + rejection_reason = db.Column( + db.String, + nullable=False, + default='', + ) #: The Event containing this registration event = db.relationship( 'Event', @@ -234,8 +237,12 @@ class Registration(db.Model): def get_all_for_event(cls, event): """Retrieve all registrations in all registration forms of an event.""" from indico.modules.events.registration.models.forms import RegistrationForm - return Registration.find_all(Registration.is_active, ~RegistrationForm.is_deleted, - RegistrationForm.event_id == event.id, _join=Registration.registration_form) + return (Registration.query + .filter(Registration.is_active, + ~RegistrationForm.is_deleted, + RegistrationForm.event_id == event.id) + .join(Registration.registration_form) + .all()) @hybrid_property def is_active(self): @@ -281,7 +288,7 @@ def locator(self): @locator.uuid def locator(self): - """A locator that uses uuid instead of id""" + """A locator that uses uuid instead of id.""" return dict(self.registration_form.locator, token=self.uuid) @property @@ -315,7 +322,7 @@ def billable_data(self): @property def full_name(self): - """Returns the user's name in 'Firstname Lastname' notation.""" + """Return the user's name in 'Firstname Lastname' notation.""" return self.get_full_name(last_name_first=False) @property @@ -323,20 +330,25 @@ def display_full_name(self): """Return the full name using the user's preferred name format.""" return format_display_full_name(session.user, self) + @property + def avatar_url(self): + """Return the url of the user's avatar.""" + return url_for('event_registration.registration_avatar', self) + @property def is_ticket_blocked(self): - """Check whether the ticket is blocked by a plugin""" + """Check whether the ticket is blocked by a plugin.""" return any(values_from_signal(signals.event.is_ticket_blocked.send(self), single_value=True)) @property def is_paid(self): - """Returns whether the registration has been paid for.""" + """Return whether the registration has been paid for.""" paid_states = {TransactionStatus.successful, TransactionStatus.pending} return self.transaction is not None and self.transaction.status in paid_states @property def payment_dt(self): - """The date/time when the registration has been paid for""" + """The date/time when the registration has been paid for.""" return self.transaction.timestamp if self.is_paid else None @property @@ -360,26 +372,26 @@ def price(self): @property def summary_data(self): - """Export registration data nested in sections and fields""" + """Export registration data nested in sections and fields.""" def _fill_from_regform(): for section in self.registration_form.sections: if not section.is_visible: continue - summary[section] = OrderedDict() + summary[section] = {} for field in section.fields: if not field.is_visible: continue summary[section][field] = field_summary[field] def _fill_from_registration(): - for field, data in field_summary.iteritems(): + for field, data in field_summary.items(): section = field.parent - summary.setdefault(section, OrderedDict()) + summary.setdefault(section, {}) if field not in summary[section]: summary[section][field] = data - summary = OrderedDict() + summary = {} field_summary = {x.field_data.field: x for x in self.data} _fill_from_regform() _fill_from_registration() @@ -399,13 +411,12 @@ def sections_with_answered_fields(self): def order_by_name(cls): return db.func.lower(cls.last_name), db.func.lower(cls.first_name), cls.friendly_id - @return_ascii def __repr__(self): return format_repr(self, 'id', 'registration_form_id', 'email', 'state', user_id=None, is_deleted=False, _text=self.full_name) def get_full_name(self, last_name_first=True, last_name_upper=False, abbrev_first_name=False): - """Returns the user's in the specified notation. + """Return the user's in the specified notation. If not format options are specified, the name is returned in the 'Lastname, Firstname' notation. @@ -448,7 +459,7 @@ def render_price_adjustment(self): return self._render_price(self.price_adjustment) def sync_state(self, _skip_moderation=True): - """Sync the state of the registration""" + """Sync the state of the registration.""" initial_state = self.state regform = self.registration_form invitation = self.invitation @@ -473,7 +484,7 @@ def sync_state(self, _skip_moderation=True): signals.event.registration_state_updated.send(self, previous_state=initial_state) def update_state(self, approved=None, paid=None, rejected=None, withdrawn=None, _skip_moderation=False): - """Update the state of the registration for a given action + """Update the state of the registration for a given action. The accepted kwargs are the possible actions. ``True`` means that the action occured and ``False`` that it was reverted. @@ -551,7 +562,7 @@ def log(self, *args, **kwargs): class RegistrationData(StoredFileMixin, db.Model): - """Data entry within a registration for a field in a registration form""" + """Data entry within a registration for a field in a registration form.""" __tablename__ = 'registration_data' __table_args__ = {'schema': 'event_registration'} @@ -645,15 +656,14 @@ def _set_file(self, file_info): file = property(fset=_set_file) del _set_file - @return_ascii def __repr__(self): - return ''.format(self.registration_id, self.field_data_id, self.data) + return f'' def _build_storage_path(self): self.registration.registration_form.assign_id() self.registration.assign_id() - path_segments = ['event', strict_unicode(self.registration.event_id), 'registrations', - strict_unicode(self.registration.registration_form.id), strict_unicode(self.registration.id)] + path_segments = ['event', strict_str(self.registration.event_id), 'registrations', + strict_str(self.registration.registration_form.id), strict_str(self.registration.id)] assert None not in path_segments # add timestamp in case someone uploads the same file again filename = '{}-{}-{}'.format(self.field_data.field_id, int(time.time()), secure_filename(self.filename, 'file')) diff --git a/indico/modules/events/registration/notifications.py b/indico/modules/events/registration/notifications.py index 0d6a737f047..3427571c422 100644 --- a/indico/modules/events/registration/notifications.py +++ b/indico/modules/events/registration/notifications.py @@ -1,16 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals from indico.core.notifications import make_email, send_email +from indico.modules.events.ical import event_to_ical from indico.modules.events.registration.models.registrations import RegistrationState from indico.util.placeholders import replace_placeholders from indico.util.signals import values_from_signal @@ -27,9 +26,9 @@ def notify_invitation(invitation, email_subject, email_body, from_address): send_email(email, invitation.registration_form.event, 'Registration', user) -def _notify_registration(registration, template, to_managers=False): +def _notify_registration(registration, template, to_managers=False, attach_rejection_reason=False): from indico.modules.events.registration.util import get_ticket_attachments - attachments = None + attachments = [] regform = registration.registration_form tickets_handled = values_from_signal(signals.event.is_ticketing_handled.send(regform), single_value=True) if (not to_managers and @@ -37,9 +36,13 @@ def _notify_registration(registration, template, to_managers=False): regform.ticket_on_email and not any(tickets_handled) and registration.state == RegistrationState.complete): - attachments = get_ticket_attachments(registration) + attachments += get_ticket_attachments(registration) + if not to_managers and registration.registration_form.attach_ical: + event_ical = event_to_ical(registration.event) + attachments.append(('event.ics', event_ical, 'text/calendar')) - template = get_template_module('events/registration/emails/{}'.format(template), registration=registration) + template = get_template_module(f'events/registration/emails/{template}', registration=registration, + attach_rejection_reason=attach_rejection_reason) to_list = registration.email if not to_managers else registration.registration_form.manager_notification_recipients from_address = registration.registration_form.sender_address if not to_managers else None mail = make_email(to_list=to_list, template=template, html=True, from_address=from_address, attachments=attachments) @@ -62,7 +65,8 @@ def notify_registration_modification(registration, notify_user=True): _notify_registration(registration, 'registration_modification_to_managers.html', to_managers=True) -def notify_registration_state_update(registration): - _notify_registration(registration, 'registration_state_update_to_registrant.html') +def notify_registration_state_update(registration, attach_rejection_reason=False): + _notify_registration(registration, 'registration_state_update_to_registrant.html', + attach_rejection_reason=attach_rejection_reason) if registration.registration_form.manager_notifications_enabled: _notify_registration(registration, 'registration_state_update_to_managers.html', to_managers=True) diff --git a/indico/modules/events/registration/placeholders/invitations.py b/indico/modules/events/registration/placeholders/invitations.py index 9948754b194..6f89c079226 100644 --- a/indico/modules/events/registration/placeholders/invitations.py +++ b/indico/modules/events/registration/placeholders/invitations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from markupsafe import Markup from indico.util.i18n import _ diff --git a/indico/modules/events/registration/placeholders/registrations.py b/indico/modules/events/registration/placeholders/registrations.py index 117c9c539b3..b767fa74627 100644 --- a/indico/modules/events/registration/placeholders/registrations.py +++ b/indico/modules/events/registration/placeholders/registrations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from markupsafe import Markup from indico.modules.events.registration.models.items import PersonalDataType @@ -72,6 +70,15 @@ def render(cls, regform, registration): return Markup('{url}').format(url=url) +class RejectionReasonPlaceholder(Placeholder): + name = 'rejection_reason' + description = _("The reason why the registration was rejected") + + @classmethod + def render(cls, regform, registration): + return registration.rejection_reason + + class FieldPlaceholder(ParametrizedPlaceholder): name = 'field' description = None @@ -101,5 +108,5 @@ def iter_param_info(cls, regform, registration): if field.personal_data_type in own_placeholder_types: continue for key, description in field.field_impl.iter_placeholder_info(): - name = unicode(field.id) if key is None else '{}:{}'.format(field.id, key) + name = str(field.id) if key is None else f'{field.id}:{key}' yield name, description diff --git a/indico/modules/events/registration/schemas.py b/indico/modules/events/registration/schemas.py index a60ee5daf2e..70844a6987f 100644 --- a/indico/modules/events/registration/schemas.py +++ b/indico/modules/events/registration/schemas.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -11,7 +11,7 @@ from indico.modules.events.registration.models.forms import RegistrationForm -class RegistrationFormPrincipalSchema(mm.ModelSchema): +class RegistrationFormPrincipalSchema(mm.SQLAlchemyAutoSchema): class Meta: model = RegistrationForm fields = ('id', 'name', 'identifier') diff --git a/indico/modules/events/registration/settings.py b/indico/modules/events/registration/settings.py index 01423d0ece3..836cfe55d34 100644 --- a/indico/modules/events/registration/settings.py +++ b/indico/modules/events/registration/settings.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.settings.converters import EnumConverter from indico.modules.designer import PageOrientation, PageSize from indico.modules.events.registration.models.items import PersonalDataType @@ -33,7 +31,7 @@ class RegistrationSettingsProxy(EventSettingsProxy): - """Store per-event registration settings""" + """Store per-event registration settings.""" def get_participant_list_columns(self, event, form=None): if form is None: @@ -42,8 +40,8 @@ def get_participant_list_columns(self, event, form=None): else: try: # The int values are automatically converted to unicode when saved as JSON - form_columns = self.get(event, 'participant_list_form_columns')[unicode(form.id)] - return map(int, form_columns) + form_columns = self.get(event, 'participant_list_form_columns')[str(form.id)] + return list(map(int, form_columns)) except (ValueError, KeyError): # No settings for this form, default to the ones for the merged form column_names = self.get_participant_list_columns(event) @@ -61,14 +59,14 @@ def set_participant_list_columns(self, event, columns, form=None): # The int values are automatically converted to unicode when saved # as JSON. Do it explicitely so that it keeps working if the # behavior changes and makes sense with the code above. - form_columns[unicode(form.id)] = columns + form_columns[str(form.id)] = columns else: - form_columns.pop(unicode(form.id), None) + form_columns.pop(str(form.id), None) self.set(event, 'participant_list_form_columns', form_columns) def get_participant_list_form_ids(self, event): # Int values are converted to str when saved as JSON - return map(int, self.get(event, 'participant_list_forms')) + return list(map(int, self.get(event, 'participant_list_forms'))) def set_participant_list_form_ids(self, event, form_ids): self.set(event, 'participant_list_forms', form_ids) diff --git a/indico/modules/events/registration/stats.py b/indico/modules/events/registration/stats.py index d0ccf80e988..d75783a4c58 100644 --- a/indico/modules/events/registration/stats.py +++ b/indico/modules/events/registration/stats.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import division, unicode_literals - from collections import defaultdict, namedtuple from itertools import chain, groupby @@ -15,15 +13,15 @@ from indico.util.i18n import _ -class StatsBase(object): +class StatsBase: def __init__(self, title, subtitle, type, **kwargs): - """Base class for registration form statistics + """Base class for registration form statistics. :param title: str -- the title for the stats box :param subtitle: str -- the subtitle for the stats box :param type: str -- the type used in Jinja to display the stats """ - super(StatsBase, self).__init__(**kwargs) + super().__init__(**kwargs) self.title = title self.subtitle = subtitle self.type = type @@ -34,7 +32,7 @@ def is_currency_shown(self): class Cell(namedtuple('Cell', ['type', 'data', 'colspan', 'classes', 'qtip'])): - """Hold data and type for a cell of a stats table""" + """Hold data and type for a cell of a stats table.""" _type_defaults = { 'str': '', @@ -80,7 +78,7 @@ def __new__(cls, type='default', data=None, colspan=1, classes=None, qtip=None): classes = [] if data is None: data = Cell._type_defaults.get(type, None) - return super(Cell, cls).__new__(cls, type, data, colspan, classes, qtip) + return super().__new__(cls, type, data, colspan, classes, qtip) class DataItem(namedtuple('DataItem', ['regs', 'attendance', 'capacity', 'billable', 'cancelled', 'price', @@ -88,7 +86,7 @@ class DataItem(namedtuple('DataItem', ['regs', 'attendance', 'capacity', 'billab def __new__(cls, regs=0, attendance=0, capacity=0, billable=False, cancelled=False, price=0, fixed_price=False, paid=0, paid_amount=0, unpaid=0, unpaid_amount=0): """ - Holds the aggregation of some data, intended for stats tables as + Hold the aggregation of some data, intended for stats tables as a aggregation from which to generate cells. :param regs: int -- number of registrant @@ -108,16 +106,16 @@ def __new__(cls, regs=0, attendance=0, capacity=0, billable=False, cancelled=Fal :param unpaid_amount: float -- amount not already paid by registrants """ - return super(DataItem, cls).__new__(cls, regs, attendance, capacity, billable, cancelled, price, fixed_price, - paid, paid_amount, unpaid, unpaid_amount) + return super().__new__(cls, regs, attendance, capacity, billable, cancelled, price, fixed_price, paid, + paid_amount, unpaid, unpaid_amount) -class FieldStats(object): - """Holds stats for a registration form field""" +class FieldStats: + """Hold stats for a registration form field.""" def __init__(self, field, **kwargs): kwargs.setdefault('type', 'table') - super(FieldStats, self).__init__(**kwargs) + super().__init__(**kwargs) self._field = field self._regitems = self._get_registration_data(field) self._choices = self._get_choices(field) @@ -133,12 +131,12 @@ def _get_choices(self, field): def _get_registration_data(self, field): registration_ids = [r.id for r in field.registration_form.active_registrations] field_data_ids = [data.id for data in field.data_versions] - return RegistrationData.find_all(RegistrationData.registration_id.in_(registration_ids), - RegistrationData.field_data_id.in_(field_data_ids), - RegistrationData.data != {}) + return RegistrationData.query.filter(RegistrationData.registration_id.in_(registration_ids), + RegistrationData.field_data_id.in_(field_data_ids), + RegistrationData.data != {}).all() def _build_data(self): - """Build data from registration data and field choices + """Build data from registration data and field choices. :returns: (dict, bool) -- the data and a boolean to indicate whether the data contains billing information or not. @@ -150,15 +148,15 @@ def _build_data(self): choices['billed'][k] = self._build_regitems_data(k, list(regitems)) for k, regitems in groupby((regitem for regitem in regitems if not regitem.price), key=self._build_key): choices['not_billed'][k] = self._build_regitems_data(k, list(regitems)) - for item in self._choices.itervalues(): + for item in self._choices.values(): key = 'billed' if item['price'] else 'not_billed' choices[key].setdefault(self._build_key(item), self._build_choice_data(item)) - for key, choice in chain(choices['billed'].iteritems(), choices['not_billed'].iteritems()): + for key, choice in chain(choices['billed'].items(), choices['not_billed'].items()): data[key[:2]].append(choice) return data, bool(choices['billed']) def get_table(self): - """Returns a table containing the stats for each item. + """Return a table containing the stats for each item. :return: dict -- A table with a list of head cells (key: `'head'`) and a list of rows (key: `'rows'`) @@ -166,7 +164,7 @@ def get_table(self): """ table = defaultdict(list) table['head'] = self._get_table_head() - for (name, id), data_items in sorted(self._data.iteritems()): + for (name, id), data_items in sorted(self._data.items()): total_regs = sum(detail.regs for detail in data_items) table['rows'].append(('single-row' if len(data_items) == 1 else 'header-row', self._get_main_row_cells(data_items, name, total_regs) + @@ -180,7 +178,7 @@ def get_table(self): return table def _get_billing_cells(self, data_items): - """Return cells with billing information from data items + """Return cells with billing information from data items. :params data_items: [DataItem] -- Data items containing billing info :returns: [Cell] -- Cells containing billing information. @@ -195,7 +193,7 @@ def _get_billing_cells(self, data_items): unpaid_amount = sum(detail.unpaid_amount for detail in data_items if detail.billable) total = paid + unpaid total_amount = paid_amount + unpaid_amount - progress = [[paid / total, unpaid / total], '{} / {}'.format(paid, total)] if total else None + progress = [[paid / total, unpaid / total], f'{paid} / {total}'] if total else None return [Cell(), Cell(type='progress-stacked', data=progress, classes=['paid-unpaid-progress']), Cell(type='currency', data=paid_amount, classes=['paid-amount', 'stick-left']), @@ -203,7 +201,7 @@ def _get_billing_cells(self, data_items): Cell(type='currency', data=total_amount)] def _get_billing_details_cells(self, detail): - """Return cells with detailed billing information + """Return cells with detailed billing information. :params item_details: DataItem -- Data items containing billing info :returns: [Cell] -- Cells containing billing information. @@ -225,7 +223,7 @@ def _get_billing_details_cells(self, detail): Cell(type='currency', data=detail.paid_amount + detail.unpaid_amount)] def _build_key(self, item): - """Return the key to sort and group field choices + """Return the key to sort and group field choices. It must include the caption and the id of the item as well as other billing information by which to aggregate. @@ -236,7 +234,7 @@ def _build_key(self, item): raise NotImplementedError def _build_regitems_data(self, key, regitems): - """Return a `DataItem` aggregating data from registration items + """Return a `DataItem` aggregating data from registration items. :param regitems: list -- list of registrations items to be aggregated :returns: DataItem -- the data aggregation @@ -245,7 +243,7 @@ def _build_regitems_data(self, key, regitems): def _build_choice_data(self, item): """ - Returns a `DataItem` containing the aggregation of an item which + Return a `DataItem` containing the aggregation of an item which is billed to the registrants. :param item: list -- item to be aggregated @@ -255,7 +253,7 @@ def _build_choice_data(self, item): def _get_table_head(self): """ - Returns a list of `Cell` corresponding to the headers of a the + Return a list of `Cell` corresponding to the headers of a the table. :returns: [Cell] -- the headers of the table. @@ -264,7 +262,7 @@ def _get_table_head(self): def _get_main_row_cells(self, item_details, choice_caption, total_regs): """ - Returns the cells of the main (header or single) row of the table. + Return the cells of the main (header or single) row of the table. Each `item` has a main row. The row is a list of `Cell` which matches the table head. @@ -280,7 +278,7 @@ def _get_main_row_cells(self, item_details, choice_caption, total_regs): def _get_sub_row_cells(self, details, total_regs): """ - Returns the cells of the sub row of the table. + Return the cells of the sub row of the table. An `item` can have a sub row. The row is a list of `Cell` which matches the table head. @@ -294,10 +292,10 @@ def _get_sub_row_cells(self, details, total_regs): class OverviewStats(StatsBase): - """Generic stats for a registration form""" + """Generic stats for a registration form.""" def __init__(self, regform): - super(OverviewStats, self).__init__(title=_("Overview"), subtitle="", type='overview') + super().__init__(title=_("Overview"), subtitle="", type='overview') self.regform = regform self.registrations = regform.active_registrations self.countries, self.num_countries = self._get_countries() @@ -313,7 +311,7 @@ def _get_countries(self): if not countries: return [], 0 # Sort by highest number of people per country then alphabetically per countries' name - countries = sorted(((val, name) for name, val in countries.iteritems()), + countries = sorted(((val, name) for name, val in countries.items()), key=lambda x: (-x[0], x[1]), reverse=True) return countries[-15:], len(countries) @@ -326,8 +324,8 @@ def _get_availibility(self): class AccommodationStats(FieldStats, StatsBase): def __init__(self, field): - super(AccommodationStats, self).__init__(title=_("Accommodation"), subtitle=field.title, field=field) - self.has_capacity = any(detail.capacity for acco_details in self._data.itervalues() + super().__init__(title=_("Accommodation"), subtitle=field.title, field=field) + self.has_capacity = any(detail.capacity for acco_details in self._data.values() for detail in acco_details if detail.capacity) def _get_occupancy(self, acco_details): @@ -337,7 +335,7 @@ def _get_occupancy(self, acco_details): if not capacity: return [Cell()] regs = sum(d.regs for d in acco_details) - return [Cell(type='progress', data=(regs / capacity, '{} / {}'.format(regs, capacity)))] + return [Cell(type='progress', data=(regs / capacity, f'{regs} / {capacity}'))] def _get_occupancy_details(self, details): if not self.has_capacity: @@ -396,7 +394,7 @@ def _get_main_row_cells(self, data_items, choice_caption, total_regs): return [ Cell(type='str', data=' ' + choice_caption, classes=['cancelled-item'] if cancelled else []), Cell(type='progress', data=((total_regs / len(active_registrations), - '{} / {}'.format(total_regs, len(active_registrations))) + f'{total_regs} / {len(active_registrations)}') if active_registrations else None)) ] + self._get_occupancy(data_items) @@ -404,6 +402,6 @@ def _get_sub_row_cells(self, data_item, total_regs): return [ Cell(type='str'), Cell(type='progress', data=((data_item.regs / total_regs, - '{} / {}'.format(data_item.regs, total_regs)) + f'{data_item.regs} / {total_regs}') if total_regs else None)), ] + self._get_occupancy_details(data_item) diff --git a/indico/modules/events/registration/templates/display/event_header.html b/indico/modules/events/registration/templates/display/event_header.html index 85bc21b763f..fc4d5ba9168 100644 --- a/indico/modules/events/registration/templates/display/event_header.html +++ b/indico/modules/events/registration/templates/display/event_header.html @@ -83,8 +83,8 @@
    {%- for participant in event.published_registrations|sort(attribute='display_full_name') -%} -
  • - {{ participant.display_full_name }} +
  • + {{ participant.display_full_name }}
    • {%- endfor -%}
diff --git a/indico/modules/events/registration/templates/display/regform_display.html b/indico/modules/events/registration/templates/display/regform_display.html index 1d4380f0a34..f102f8439c5 100644 --- a/indico/modules/events/registration/templates/display/regform_display.html +++ b/indico/modules/events/registration/templates/display/regform_display.html @@ -109,7 +109,7 @@
{% block regform %}
{% block regform %}
{% trans %}Modify{% endtrans %} diff --git a/indico/modules/events/registration/templates/emails/base_registration_details.html b/indico/modules/events/registration/templates/emails/base_registration_details.html index 4f108c45176..fe44bc93fda 100644 --- a/indico/modules/events/registration/templates/emails/base_registration_details.html +++ b/indico/modules/events/registration/templates/emails/base_registration_details.html @@ -1,11 +1,11 @@ {% extends 'events/registration/emails/base_registration.html' %} {% block registration_body %} - {% for section, fields in registration.summary_data.iteritems() if not section.is_manager_only %} + {% for section, fields in registration.summary_data.items() if not section.is_manager_only %}

{{ section.title }}

- {% for field, regdata in fields.iteritems() if regdata.friendly_data %} + {% for field, regdata in fields.items() if regdata.friendly_data %} {{ render_field(field) }} {{ render_regdata(regdata) }} {% endfor %} diff --git a/indico/modules/events/registration/templates/emails/registration_creation_to_managers.html b/indico/modules/events/registration/templates/emails/registration_creation_to_managers.html index 3a837c39c3d..2b9d16a3430 100644 --- a/indico/modules/events/registration/templates/emails/registration_creation_to_managers.html +++ b/indico/modules/events/registration/templates/emails/registration_creation_to_managers.html @@ -31,6 +31,13 @@ {%- endmacro %} +{% macro render_rejection_reason() %} + {% if registration.state.name == 'rejected' and registration.rejection_reason %} +
Rejection reason: {{ registration.rejection_reason }}
+ {% endif %} +{% endmacro %} + + {% macro render_text_pending() %} {% if registration.state.name == 'pending' %} Please note that it's waiting for manual approval by a manager. diff --git a/indico/modules/events/registration/templates/emails/registration_creation_to_registrant.html b/indico/modules/events/registration/templates/emails/registration_creation_to_registrant.html index e3bede8bef8..23a92d213d8 100644 --- a/indico/modules/events/registration/templates/emails/registration_creation_to_registrant.html +++ b/indico/modules/events/registration/templates/emails/registration_creation_to_registrant.html @@ -16,15 +16,13 @@ {{ render_text_unpaid() }} {% endblock %}

-

- {% if registration.state.name == 'pending' %} - {{ registration.registration_form.message_pending | escape | urlize }} - {% elif registration.state.name == 'unpaid' %} - {{ registration.registration_form.message_unpaid | escape | urlize }} - {% elif registration.state.name == 'complete' %} - {{ registration.registration_form.message_complete | escape | urlize }} - {% endif %} -

+ {% if registration.state.name == 'pending' %} + {{ registration.registration_form.message_pending | markdown }} + {% elif registration.state.name == 'unpaid' %} + {{ registration.registration_form.message_unpaid | markdown }} + {% elif registration.state.name == 'complete' %} + {{ registration.registration_form.message_complete | markdown }} + {% endif %} {%- endblock %} {% block registration_footer %} @@ -51,6 +49,13 @@ {% endmacro %} +{% macro render_rejection_reason() %} + {% if registration.state.name == 'rejected' and registration.rejection_reason %} +
Rejection reason: {{ registration.rejection_reason }}
+ {% endif %} +{% endmacro %} + + {% macro render_text_pending() %} {% if registration.state.name == 'pending' %} Please note that this event requires manual approval. diff --git a/indico/modules/events/registration/templates/emails/registration_state_update_to_managers.html b/indico/modules/events/registration/templates/emails/registration_state_update_to_managers.html index 88d3988b09d..5e4900ed694 100644 --- a/indico/modules/events/registration/templates/emails/registration_state_update_to_managers.html +++ b/indico/modules/events/registration/templates/emails/registration_state_update_to_managers.html @@ -8,6 +8,7 @@ {% block registration_header_text %} The registration {{ render_registration_info() }} is now {{ registration.state.title|lower }}. + {{ render_rejection_reason() }} {{ render_text_pending() }} {{ render_text_manage() }} {% endblock %} diff --git a/indico/modules/events/registration/templates/emails/registration_state_update_to_registrant.html b/indico/modules/events/registration/templates/emails/registration_state_update_to_registrant.html index bedf44837a2..f5e7d83af0f 100644 --- a/indico/modules/events/registration/templates/emails/registration_state_update_to_registrant.html +++ b/indico/modules/events/registration/templates/emails/registration_state_update_to_registrant.html @@ -8,6 +8,7 @@ {% block registration_header_text -%} Your registration {{ render_registration_info() }} is now {{ registration.state.title|lower }}. + {{ render_rejection_reason() if attach_rejection_reason }} {{ render_text_pending() }} {{ render_text_unpaid() }} {%- endblock %} diff --git a/indico/modules/events/registration/templates/management/_registration_details.html b/indico/modules/events/registration/templates/management/_registration_details.html index 63072b683c5..deb5dd669d1 100644 --- a/indico/modules/events/registration/templates/management/_registration_details.html +++ b/indico/modules/events/registration/templates/management/_registration_details.html @@ -105,10 +105,9 @@
+ data-title="{% trans %}Reject registration{% endtrans %}" + data-ajax-dialog> {%- trans %}Reject{% endtrans -%}
@@ -184,6 +183,9 @@ This registration is {{ state }} {%- endtrans %}
+ {% if registration.state.name == 'rejected' and registration.rejection_reason %} + {% trans reason=registration.rejection_reason %}Reason: {{ reason }}{% endtrans %} + {% endif %}
{% set action = _('Reset rejection') if registration.state.name == 'rejected' else _('Reset withdrawal') %} diff --git a/indico/modules/events/registration/templates/management/_reglist.html b/indico/modules/events/registration/templates/management/_reglist.html index ed555e5421c..68b7e4055e8 100644 --- a/indico/modules/events/registration/templates/management/_reglist.html +++ b/indico/modules/events/registration/templates/management/_reglist.html @@ -81,7 +81,7 @@ {% endif %} {% else %} -
+ {% endif %} {% endfor %} {% for item in dynamic_columns %} diff --git a/indico/modules/events/registration/templates/management/regform_display.html b/indico/modules/events/registration/templates/management/regform_display.html index 9596e72e619..39a73e90ac8 100644 --- a/indico/modules/events/registration/templates/management/regform_display.html +++ b/indico/modules/events/registration/templates/management/regform_display.html @@ -84,7 +84,7 @@ classes="i-box titled disable-if-locked") %} {% endcall %} diff --git a/indico/modules/events/registration/templates/management/regform_modify.html b/indico/modules/events/registration/templates/management/regform_modify.html index 8ce19324c1c..afdcf52564d 100644 --- a/indico/modules/events/registration/templates/management/regform_modify.html +++ b/indico/modules/events/registration/templates/management/regform_modify.html @@ -10,7 +10,7 @@
{% block regform %}
  • {%- trans %}Approve registrations{% endtrans -%}
  • - + data-title="{% trans %}Reject registrations{% endtrans %}" + data-update="#registration-list" + data-ajax-dialog> {%- trans %}Reject registrations{% endtrans -%}
    • diff --git a/indico/modules/events/registration/templates/management/regform_stats.html b/indico/modules/events/registration/templates/management/regform_stats.html index bc6c078a995..2d5e08bcf96 100644 --- a/indico/modules/events/registration/templates/management/regform_stats.html +++ b/indico/modules/events/registration/templates/management/regform_stats.html @@ -63,7 +63,7 @@
    {% trans %}Registrants per country{% endtrans %}
    -
    +
    -{% endblock %} diff --git a/indico/modules/events/sessions/util.py b/indico/modules/events/sessions/util.py index c260d3b48a7..ef960113906 100644 --- a/indico/modules/events/sessions/util.py +++ b/indico/modules/events/sessions/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import defaultdict from io import BytesIO @@ -29,7 +27,7 @@ def can_manage_sessions(user, event, permission=None): - """Check whether a user can manage any sessions in an event""" + """Check whether a user can manage any sessions in an event.""" if event.can_manage(user): return True return any(s.can_manage(user, permission) @@ -71,16 +69,16 @@ def getBody(self, story=None): rows = [] row_values = [] for col in [_('ID'), _('Type'), _('Title'), _('Code'), _('Description')]: - row_values.append(Paragraph('{}'.format(col), text_style)) + row_values.append(Paragraph(f'{col}', text_style)) rows.append(row_values) for sess in self.sessions: rows.append([ Paragraph(sess.friendly_id, text_style), - Paragraph(sess.type.name.encode('utf-8') if sess.type else '', text_style), - Paragraph(sess.title.encode('utf-8'), text_style), - Paragraph(sess.code.encode('utf-8'), text_style), - Paragraph(sess.description.encode('utf-8'), text_style) + Paragraph(sess.type.name if sess.type else '', text_style), + Paragraph(sess.title, text_style), + Paragraph(sess.code, text_style), + Paragraph(sess.description, text_style) ]) col_widths = (None,) * 5 @@ -95,7 +93,7 @@ def getBody(self, story=None): def generate_pdf_from_sessions(sessions): - """Generate a PDF file from a given session list""" + """Generate a PDF file from a given session list.""" pdf = SessionListToPDF(sessions) return BytesIO(pdf.getPDFBin()) @@ -116,8 +114,9 @@ def session_coordinator_priv_enabled(event, priv): def get_events_with_linked_sessions(user, dt=None): - """Returns a dict with keys representing event_id and the values containing - data about the user rights for sessions within the event + """ + Return a dict with keys representing event_id and the values containing + data about the user rights for sessions within the event. :param user: A `User` :param dt: Only include events taking place on/after that date @@ -178,15 +177,8 @@ def serialize_session_for_ical(sess): } -def get_session_ical_file(sess): - from indico.web.http_api.metadata.serializer import Serializer - data = {'results': serialize_session_for_ical(sess) if sess.start_dt and sess.end_dt else []} - serializer = Serializer.create('ics') - return BytesIO(serializer(data)) - - def get_session_timetable_pdf(sess, **kwargs): - from indico.legacy.pdfinterface.conference import TimeTablePlain, TimetablePDFFormat + from indico.legacy.pdfinterface.conference import TimetablePDFFormat, TimeTablePlain pdf_format = TimetablePDFFormat(params={'coverPage': False}) return TimeTablePlain(sess.event, session.user, showSessions=[sess.id], showDays=[], sortingCrit=None, ttPDFFormat=pdf_format, pagesize='A4', fontsize='normal', diff --git a/indico/modules/events/sessions/views.py b/indico/modules/events/sessions/views.py index f3c7dfddb4b..ac177662c22 100644 --- a/indico/modules/events/sessions/views.py +++ b/indico/modules/events/sessions/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement from indico.modules.events.views import WPConferenceDisplayBase diff --git a/indico/modules/events/settings.py b/indico/modules/events/settings.py index 362305b04b4..0aa421f9b35 100644 --- a/indico/modules/events/settings.py +++ b/indico/modules/events/settings.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import os import re from functools import wraps @@ -36,7 +34,7 @@ def wrapper(self, event, *args, **kwargs): class EventACLProxy(ACLProxyBase): - """Proxy class for event-specific ACL settings""" + """Proxy class for event-specific ACL settings.""" @event_or_id def get(self, event, name): @@ -50,7 +48,7 @@ def get(self, event, name): @event_or_id def set(self, event, name, acl): - """Replaces an ACL with a new one + """Replace an ACL with a new one. :param event: Event (or its ID) :param name: Setting name @@ -62,7 +60,7 @@ def set(self, event, name, acl): @event_or_id def contains_user(self, event, name, user): - """Checks if a user is in an ACL. + """Check if a user is in an ACL. To pass this check, the user can either be in the ACL itself or in a group in the ACL. @@ -75,7 +73,7 @@ def contains_user(self, event, name, user): @event_or_id def add_principal(self, event, name, principal): - """Adds a principal to an ACL + """Add a principal to an ACL. :param event: Event (or its ID) :param name: Setting name @@ -87,7 +85,7 @@ def add_principal(self, event, name, principal): @event_or_id def remove_principal(self, event, name, principal): - """Removes a principal from an ACL + """Remove a principal from an ACL. :param event: Event (or its ID) :param name: Setting name @@ -98,24 +96,24 @@ def remove_principal(self, event, name, principal): self._flush_cache() def merge_users(self, target, source): - """Replaces all ACL user entries for `source` with `target`""" + """Replace all ACL user entries for `source` with `target`.""" EventSettingPrincipal.merge_users(self.module, target, source) self._flush_cache() class EventSettingsProxy(SettingsProxyBase): - """Proxy class to access event-specific settings for a certain module""" + """Proxy class to access event-specific settings for a certain module.""" acl_proxy_class = EventACLProxy @property def query(self): - """Returns a query object filtering by the proxy's module.""" - return EventSetting.find(module=self.module) + """Return a query object filtering by the proxy's module.""" + return EventSetting.query.filter_by(module=self.module) @event_or_id def get_all(self, event, no_defaults=False): - """Retrieves all settings + """Retrieve all settings. :param event: Event (or its ID) :param no_defaults: Only return existing settings and ignore defaults. @@ -125,7 +123,7 @@ def get_all(self, event, no_defaults=False): @event_or_id def get(self, event, name, default=SettingsProxyBase.default_sentinel): - """Retrieves the value of a single setting. + """Retrieve the value of a single setting. :param event: Event (or its ID) :param name: Setting name @@ -137,7 +135,7 @@ def get(self, event, name, default=SettingsProxyBase.default_sentinel): @event_or_id def set(self, event, name, value): - """Sets a single setting. + """Set a single setting. :param event: Event (or its ID) :param name: Setting name @@ -149,12 +147,12 @@ def set(self, event, name, value): @event_or_id def set_multi(self, event, items): - """Sets multiple settings at once. + """Set multiple settings at once. :param event: Event (or its ID) :param items: Dict containing the new settings """ - items = {k: self._convert_from_python(k, v) for k, v in items.iteritems()} + items = {k: self._convert_from_python(k, v) for k, v in items.items()} self._split_call(items, lambda x: EventSetting.set_multi(self.module, x, event_id=event), lambda x: EventSettingPrincipal.set_acl_multi(self.module, x, event_id=event)) @@ -162,7 +160,7 @@ def set_multi(self, event, items): @event_or_id def delete(self, event, *names): - """Deletes settings. + """Delete settings. :param event: Event (or its ID) :param names: One or more names of settings to delete @@ -174,7 +172,7 @@ def delete(self, event, *names): @event_or_id def delete_all(self, event): - """Deletes all settings. + """Delete all settings. :param event: Event (or its ID) """ @@ -187,7 +185,7 @@ class EventSettingProperty(SettingProperty): attr = 'event' -class ThemeSettingsProxy(object): +class ThemeSettingsProxy: @property @memoize def settings(self): @@ -204,7 +202,7 @@ def settings(self): with open(path) as f: data = f.read() settings = {k: v - for k, v in yaml.safe_load(core_data + '\n' + data).viewitems() + for k, v in yaml.safe_load(core_data + '\n' + data).items() if not k.startswith('__core_')} # We assume there's no more than one theme plugin that provides defaults. # If that's not the case the last one "wins". We could reject this but it @@ -215,7 +213,7 @@ def settings(self): # to avoid using definition names that are likely to cause collisions. # Either way, if someone does this on purpose chances are good they want # to override a default style so let them do so... - for name, definition in settings.get('definitions', {}).viewitems(): + for name, definition in settings.get('definitions', {}).items(): definition['plugin'] = plugin definition.setdefault('user_visible', False) core_settings['definitions'][name] = definition @@ -233,7 +231,7 @@ def defaults(self): @memoize def get_themes_for(self, event_type): - return {theme_id: theme_data for theme_id, theme_data in self.themes.viewitems() + return {theme_id: theme_data for theme_id, theme_data in self.themes.items() if event_type in theme_data['event_types']} diff --git a/indico/modules/events/static/__init__.py b/indico/modules/events/static/__init__.py index f8acfb9c8e4..9d5e22a1633 100644 --- a/indico/modules/events/static/__init__.py +++ b/indico/modules/events/static/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals @@ -22,7 +20,7 @@ @signals.users.merged.connect def _merge_users(target, source, **kwargs): - StaticSite.find(creator_id=source.id).update({StaticSite.creator_id: target.id}) + StaticSite.query.filter_by(creator_id=source.id).update({StaticSite.creator_id: target.id}) @signals.menu.items.connect_via('event-management-sidemenu') diff --git a/indico/modules/events/static/blueprint.py b/indico/modules/events/static/blueprint.py index d1e03e80200..8c895798d20 100644 --- a/indico/modules/events/static/blueprint.py +++ b/indico/modules/events/static/blueprint.py @@ -1,18 +1,16 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.static.controllers import RHStaticSiteBuild, RHStaticSiteDownload, RHStaticSiteList from indico.web.flask.wrappers import IndicoBlueprint _bp = IndicoBlueprint('static_site', __name__, template_folder='templates', virtual_template_folder='events/static', - url_prefix='/event//manage/offline-copy') + url_prefix='/event//manage/offline-copy') # Event management _bp.add_url_rule('/', 'list', RHStaticSiteList) diff --git a/indico/modules/events/static/controllers.py b/indico/modules/events/static/controllers.py index 3a2acf22383..ead0f94ff7b 100644 --- a/indico/modules/events/static/controllers.py +++ b/indico/modules/events/static/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import redirect, request, session from werkzeug.exceptions import NotFound diff --git a/indico/modules/events/static/models/static.py b/indico/modules/events/static/models/static.py index cc55c2d5848..25c08b20ec5 100644 --- a/indico/modules/events/static/models/static.py +++ b/indico/modules/events/static/models/static.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import posixpath from indico.core.config import config @@ -14,9 +12,9 @@ from indico.core.db.sqlalchemy import PyIntEnum, UTCDateTime from indico.core.storage import StoredFileMixin from indico.util.date_time import now_utc +from indico.util.enum import RichIntEnum from indico.util.i18n import _ -from indico.util.string import format_repr, return_ascii, strict_unicode -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_repr, strict_str class StaticSiteState(RichIntEnum): @@ -91,15 +89,14 @@ class StaticSite(StoredFileMixin, db.Model): @property def locator(self): - return {'confId': self.event_id, 'id': self.id} + return {'event_id': self.event_id, 'id': self.id} def _build_storage_path(self): - path_segments = ['event', strict_unicode(self.event.id), 'static'] + path_segments = ['event', strict_str(self.event.id), 'static'] self.assign_id() - filename = '{}-{}'.format(self.id, self.filename) + filename = f'{self.id}-{self.filename}' path = posixpath.join(*(path_segments + [filename])) return config.STATIC_SITE_STORAGE, path - @return_ascii def __repr__(self): return format_repr(self, 'id', 'event_id', 'state') diff --git a/indico/modules/events/static/offline.py b/indico/modules/events/static/offline.py index 2fc6516cd18..150b5ada40c 100644 --- a/indico/modules/events/static/offline.py +++ b/indico/modules/events/static/offline.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -32,13 +32,14 @@ from indico.modules.events.contributions.controllers.display import (RHAuthorList, RHContributionAuthor, RHContributionDisplay, RHContributionList, RHSpeakerList, RHSubcontributionDisplay) -from indico.modules.events.contributions.util import get_contribution_ical_file +from indico.modules.events.contributions.ical import contribution_to_ical from indico.modules.events.layout.models.menu import MenuEntryType from indico.modules.events.layout.util import menu_entries_for_event from indico.modules.events.models.events import EventType from indico.modules.events.registration.controllers.display import RHParticipantList from indico.modules.events.sessions.controllers.display import RHDisplaySession -from indico.modules.events.sessions.util import get_session_ical_file, get_session_timetable_pdf +from indico.modules.events.sessions.ical import session_to_ical +from indico.modules.events.sessions.util import get_session_timetable_pdf from indico.modules.events.static.util import collect_static_files, override_request_endpoint, rewrite_css_urls from indico.modules.events.timetable.controllers.display import RHTimetable from indico.modules.events.timetable.util import get_timetable_offline_pdf_generator @@ -53,9 +54,9 @@ def create_static_site(rh, event): """Create a static (offline) version of an Indico event. - :param rh: Request handler object - :param event: Event in question - :return: Path to the resulting ZIP file + :param rh: Request handler object + :param event: Event in question + :return: Path to the resulting ZIP file """ try: g.static_site = True @@ -71,7 +72,7 @@ def _normalize_path(path): return secure_filename(strip_tags(path)) -class StaticEventCreator(object): +class StaticEventCreator: """Define process which generates a static (offline) version of an Indico event.""" def __init__(self, rh, event): @@ -79,18 +80,19 @@ def __init__(self, rh, event): self.event = event self._display_tz = self.event.display_tzinfo.zone self._zip_file = None - self._content_dir = _normalize_path(u'OfflineWebsite-{}'.format(event.title)) + self._content_dir = _normalize_path(f'OfflineWebsite-{event.title}') self._web_dir = os.path.join(get_root_path('indico'), 'web') self._static_dir = os.path.join(self._web_dir, 'static') def create(self): """Trigger the creation of a ZIP file containing the site.""" - temp_file = NamedTemporaryFile(suffix='indico.tmp', dir=config.TEMP_DIR) + temp_file = NamedTemporaryFile(prefix=f'static-{self.event.id}-', suffix='.zip', dir=config.TEMP_DIR, + delete=False) self._zip_file = ZipFile(temp_file.name, 'w', allowZip64=True) with collect_static_files() as used_assets: # create the home page html - html = self._create_home().encode('utf-8') + html = self._create_home() # Mathjax plugins can only be known in runtime self._copy_folder(os.path.join(self._content_dir, 'static', 'dist', 'js', 'mathjax'), @@ -112,17 +114,15 @@ def create(self): if config.CUSTOMIZATION_DIR: self._copy_customization_files(used_assets) - temp_file.delete = False chmod_umask(temp_file.name) self._zip_file.close() return temp_file.name def _write_generated_js(self): - global_js = generate_global_file().encode('utf-8') - user_js = generate_user_file().encode('utf-8') - i18n_js = u"window.TRANSLATIONS = {};".format(generate_i18n_file(session.lang)).encode('utf-8') - react_i18n_js = u"window.REACT_TRANSLATIONS = {};".format( - generate_i18n_file(session.lang, react=True)).encode('utf-8') + global_js = generate_global_file() + user_js = generate_user_file() + i18n_js = f"window.TRANSLATIONS = {generate_i18n_file(session.lang)};" + react_i18n_js = f"window.REACT_TRANSLATIONS = {generate_i18n_file(session.lang, react=True)};" gen_path = os.path.join(self._content_dir, 'assets') self._zip_file.writestr(os.path.join(gen_path, 'js-vars', 'global.js'), global_js) self._zip_file.writestr(os.path.join(gen_path, 'js-vars', 'user.js'), user_js) @@ -206,7 +206,7 @@ def _add_material(self, target, type_): continue if attachment.type == AttachmentType.file: dst_path = posixpath.join(self._content_dir, "material", type_, - "{}-{}".format(attachment.id, attachment.file.filename)) + f"{attachment.id}-{attachment.file.filename}") with attachment.file.get_local_path() as file_path: self._copy_file(dst_path, file_path) @@ -224,7 +224,7 @@ def _copy_folder(self, dest, src): class StaticConferenceCreator(StaticEventCreator): def __init__(self, rh, event): - super(StaticConferenceCreator, self).__init__(rh, event) + super().__init__(rh, event) # Menu entries we want to include in the offline version. # Those which are backed by a WP class get their name from that class; # the others are simply hardcoded. @@ -240,7 +240,7 @@ def __init__(self, rh, event): RHTimetable: WPStaticTimetable, RHDisplayTracks: WPStaticConferenceProgram } - for rh_cls, wp in rhs.viewitems(): + for rh_cls, wp in rhs.items(): rh = rh_cls() rh.view_class = wp if rh_cls is RHTimetable: @@ -255,7 +255,7 @@ def _create_home(self): for image_file in used_images: with image_file.open() as f: self._zip_file.writestr(os.path.join(self._content_dir, - 'images/{}-{}'.format(image_file.id, image_file.filename)), + f'images/{image_file.id}-{image_file.filename}'), f.read()) if self.event.has_logo: self._zip_file.writestr(os.path.join(self._content_dir, 'logo.png'), self.event.logo) @@ -298,7 +298,7 @@ def _get_menu_items(self): def _get_builtin_page(self, entry): obj = self._menu_offline_items.get(entry.name) if isinstance(obj, RH): - request.view_args = {'confId': self.event.id} + request.view_args = {'event_id': self.event.id} with override_request_endpoint(obj.view_class.endpoint): obj._process_args() self._add_page(obj._process(), obj.view_class.endpoint, self.event) @@ -312,7 +312,7 @@ def _get_custom_page(self, page): self._add_page(html, 'event_pages.page_display', page) def _get_url(self, uh_or_endpoint, target, **params): - if isinstance(uh_or_endpoint, basestring): + if isinstance(uh_or_endpoint, str): return url_for(uh_or_endpoint, target, **params) else: return str(uh_or_endpoint.getStaticURL(target, **params)) @@ -320,7 +320,7 @@ def _get_url(self, uh_or_endpoint, target, **params): def _add_page(self, html, uh_or_endpoint, target=None, **params): url = self._get_url(uh_or_endpoint, target, **params) fname = os.path.join(self._content_dir, url) - self._zip_file.writestr(fname, html.encode('utf-8')) + self._zip_file.writestr(fname, html) def _add_from_rh(self, rh_class, view_class, params, url_for_target): rh = rh_class() @@ -333,7 +333,7 @@ def _add_from_rh(self, rh_class, view_class, params, url_for_target): def _get_contrib(self, contrib): self._add_from_rh(RHContributionDisplay, WPStaticContributionDisplay, - {'confId': self.event.id, 'contrib_id': contrib.id}, + {'event_id': self.event.id, 'contrib_id': contrib.id}, contrib) if config.LATEX_ENABLED: self._add_pdf(contrib, 'contributions.export_pdf', ContribToPDF, contrib=contrib) @@ -344,16 +344,16 @@ def _get_contrib(self, contrib): self._get_author(contrib, author) if contrib.timetable_entry: - self._add_file(get_contribution_ical_file(contrib), 'contributions.export_ics', contrib) + self._add_file(contribution_to_ical(contrib), 'contributions.export_ics', contrib) def _get_sub_contrib(self, subcontrib): self._add_from_rh(RHSubcontributionDisplay, WPStaticSubcontributionDisplay, - {'confId': self.event.id, 'contrib_id': subcontrib.contribution.id, + {'event_id': self.event.id, 'contrib_id': subcontrib.contribution.id, 'subcontrib_id': subcontrib.id}, subcontrib) def _get_author(self, contrib, author): rh = RHContributionAuthor() - params = {'confId': self.event.id, 'contrib_id': contrib.id, 'person_id': author.id} + params = {'event_id': self.event.id, 'contrib_id': contrib.id, 'person_id': author.id} request.view_args = params with override_request_endpoint('contributions.display_author'): rh._process_args() @@ -362,12 +362,12 @@ def _get_author(self, contrib, author): def _get_session(self, session): self._add_from_rh(RHDisplaySession, WPStaticSessionDisplay, - {'confId': self.event.id, 'session_id': session.id}, session) + {'event_id': self.event.id, 'session_id': session.id}, session) pdf = get_session_timetable_pdf(session, tz=self._display_tz) self._add_pdf(session, 'sessions.export_session_timetable', pdf) - self._add_file(get_session_ical_file(session), 'sessions.export_ics', session) + self._add_file(session_to_ical(session), 'sessions.export_ics', session) def _add_pdf(self, target, uh_or_endpoint, generator_class_or_instance, **kwargs): if inspect.isclass(generator_class_or_instance): @@ -379,11 +379,11 @@ def _add_pdf(self, target, uh_or_endpoint, generator_class_or_instance, **kwargs # Got legacy reportlab PDF generator instead of the LaTex-based one self._add_file(pdf.getPDFBin(), uh_or_endpoint, target) else: - with open(pdf.generate()) as f: + with open(pdf.generate(), 'rb') as f: self._add_file(f, uh_or_endpoint, target) def _add_file(self, file_like_or_str, uh_or_endpoint, target): - if isinstance(file_like_or_str, str): + if isinstance(file_like_or_str, (str, bytes)): content = file_like_or_str else: content = file_like_or_str.read() diff --git a/indico/modules/events/static/tasks.py b/indico/modules/events/static/tasks.py index 61db3f706fa..b552a7c75b5 100644 --- a/indico/modules/events/static/tasks.py +++ b/indico/modules/events/static/tasks.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import timedelta from celery.schedules import crontab @@ -37,7 +35,7 @@ def build_static_site(static_site): zip_file_path = create_static_site(rh, static_site.event) static_site.state = StaticSiteState.success static_site.content_type = 'application/zip' - static_site.filename = 'offline_site_{}.zip'.format(static_site.event.id) + static_site.filename = f'offline_site_{static_site.event.id}.zip' with open(zip_file_path, 'rb') as f: static_site.save(f) db.session.commit() @@ -61,12 +59,14 @@ def notify_static_site_success(static_site): @celery.periodic_task(name='static_sites_cleanup', run_every=crontab(minute='30', hour='3', day_of_week='monday')) def static_sites_cleanup(days=30): - """Clean up old static sites + """Clean up old static sites. :param days: number of days after which to remove static sites """ - expired_sites = StaticSite.find_all(StaticSite.requested_dt < (now_utc() - timedelta(days=days)), - StaticSite.state == StaticSiteState.success) + expired_sites = (StaticSite.query + .filter(StaticSite.requested_dt < (now_utc() - timedelta(days=days)), + StaticSite.state == StaticSiteState.success) + .all()) logger.info('Removing %d expired static sites from the past %d days', len(expired_sites), days) try: for site in expired_sites: diff --git a/indico/modules/events/static/util.py b/indico/modules/events/static/util.py index a059ccf0e81..17b785260ec 100644 --- a/indico/modules/events/static/util.py +++ b/indico/modules/events/static/util.py @@ -1,17 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import base64 import mimetypes import re -import urlparse from contextlib import contextmanager +from urllib.parse import urlsplit, urlunsplit import requests from flask import current_app, g, request @@ -54,7 +52,7 @@ def _create_data_uri(url, filename): data = base64.b64encode(response.content) content_type = (mimetypes.guess_type(filename)[0] or response.headers.get('Content-Type', 'application/octet-stream')) - return 'data:{};base64,{}'.format(content_type, data) + return f'data:{content_type};base64,{data}' def _rewrite_event_asset_url(event, url): @@ -62,7 +60,7 @@ def _rewrite_event_asset_url(event, url): Only assets contained within the event will be taken into account """ - scheme, netloc, path, qs, anchor = urlparse.urlsplit(url) + scheme, netloc, path, qs, anchor = urlsplit(url) netloc = netloc or current_app.config['SERVER_NAME'] scheme = scheme or 'https' @@ -73,20 +71,20 @@ def _rewrite_event_asset_url(event, url): endpoint_info = endpoint_for_url(path) if endpoint_info: endpoint, data = endpoint_info - if endpoint == 'event_images.image_display' and int(data['confId']) == event.id: + if endpoint == 'event_images.image_display' and data['event_id'] == event.id: image_file = ImageFile.get(data['image_id']) if image_file and image_file.event == event: - return 'images/{}-{}'.format(image_file.id, image_file.filename), image_file + return f'images/{image_file.id}-{image_file.filename}', image_file # if the URL is not internal or just not an image, # we embed the contents using a data URI - data_uri = _create_data_uri(urlparse.urlunsplit((scheme, netloc, path, qs, '')), urlparse.urlsplit(path)[-1]) + data_uri = _create_data_uri(urlunsplit((scheme, netloc, path, qs, '')), urlsplit(path)[-1]) return data_uri, None def _remove_anchor(url): """Remove the anchor from a URL.""" - scheme, netloc, path, qs, anchor = urlparse.urlsplit(url) - return urlparse.urlunsplit((scheme, netloc, path, qs, '')) + scheme, netloc, path, qs, anchor = urlsplit(url) + return urlunsplit((scheme, netloc, path, qs, '')) def rewrite_css_urls(event, css): @@ -102,18 +100,18 @@ def _replace_url(m): rewritten_url, image_file = _rewrite_event_asset_url(event, prefix + url) if image_file: used_images.add(image_file) - return 'url({})'.format(rewritten_url) + return f'url({rewritten_url})' else: rewritten_url = rewrite_static_url(url) used_urls.add(_remove_anchor(rewritten_url)) if url.startswith('/static/plugins/'): - return "url('../../../../../{}')".format(rewritten_url) + return f"url('../../../../../{rewritten_url}')" else: - return "url('../../../{}')".format(rewritten_url) + return f"url('../../../{rewritten_url}')" indico_path = url_parse(config.BASE_URL).path - new_css = re.sub(_css_url_pattern.format(indico_path), _replace_url, css.decode('utf-8'), flags=re.MULTILINE) - return new_css.encode('utf-8'), used_urls, used_images + new_css = re.sub(_css_url_pattern.format(indico_path), _replace_url, css, flags=re.MULTILINE) + return new_css, used_urls, used_images def url_to_static_filename(endpoint, url): @@ -130,7 +128,7 @@ def url_to_static_filename(endpoint, url): url = rewrite_static_url(url) else: # get rid of [/whatever]/event/1234 - url = re.sub(r'{}(?:/event/\d+)?/(.*)'.format(indico_path), r'\1', url) + url = re.sub(fr'{indico_path}(?:/event/\d+)?/(.*)', r'\1', url) if not url.startswith('assets/'): # replace all remaining slashes url = url.rstrip('/').replace('/', '--') @@ -160,9 +158,9 @@ class RewrittenManifest(Manifest): """A manifest that rewrites its asset paths.""" def __init__(self, manifest): - super(RewrittenManifest, self).__init__() + super().__init__() self._entries = {k: JinjaManifestEntry(entry.name, self._rewrite_paths(entry._paths)) - for k, entry in manifest._entries.viewitems()} + for k, entry in manifest._entries.items()} self.used_assets = set() def _rewrite_paths(self, paths): @@ -170,7 +168,7 @@ def _rewrite_paths(self, paths): def __getitem__(self, key): self.used_assets.add(key) - return super(RewrittenManifest, self).__getitem__(key) + return super().__getitem__(key) @contextmanager @@ -180,7 +178,7 @@ def collect_static_files(): g.used_url_for_assets = set() used_assets = set() yield used_assets - for manifest in g.custom_manifests.viewvalues(): + for manifest in g.custom_manifests.values(): used_assets |= {p for k in manifest.used_assets for p in manifest[k]._paths} used_assets |= {rewrite_static_url(url) for url in g.used_url_for_assets} del g.custom_manifests diff --git a/indico/modules/events/static/views.py b/indico/modules/events/static/views.py index 8eaefc1d4ac..6fd83c29a12 100644 --- a/indico/modules/events/static/views.py +++ b/indico/modules/events/static/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement diff --git a/indico/modules/events/surveys/__init__.py b/indico/modules/events/surveys/__init__.py index 97a90db207c..ab928f3bee2 100644 --- a/indico/modules/events/surveys/__init__.py +++ b/indico/modules/events/surveys/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import render_template, session from indico.core import signals @@ -28,7 +26,7 @@ @signals.users.merged.connect def _merge_users(target, source, **kwargs): from indico.modules.events.surveys.models.submissions import SurveySubmission - SurveySubmission.find(user_id=source.id).update({SurveySubmission.user_id: target.id}) + SurveySubmission.query.filter_by(user_id=source.id).update({SurveySubmission.user_id: target.id}) @signals.menu.items.connect_via('event-management-sidemenu') diff --git a/indico/modules/events/surveys/blueprint.py b/indico/modules/events/surveys/blueprint.py index fd3bad21b46..39bfd1dc81c 100644 --- a/indico/modules/events/surveys/blueprint.py +++ b/indico/modules/events/surveys/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.surveys.controllers.display import RHSaveSurveyAnswers, RHSubmitSurvey, RHSurveyList from indico.modules.events.surveys.controllers.management.questionnaire import (RHAddSurveyQuestion, RHAddSurveySection, RHAddSurveyText, RHDeleteSurveyQuestion, @@ -29,7 +27,7 @@ _bp = IndicoBlueprint('surveys', __name__, template_folder='templates', virtual_template_folder='events/surveys', - url_prefix='/event/', event_feature='surveys') + url_prefix='/event/', event_feature='surveys') # survey display/submission _bp.add_url_rule('/surveys/', 'display_survey_list', RHSurveyList) diff --git a/indico/modules/events/surveys/client/js/index.js b/indico/modules/events/surveys/client/js/index.js index ccc21c14aa5..17913692091 100644 --- a/indico/modules/events/surveys/client/js/index.js +++ b/indico/modules/events/surveys/client/js/index.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/events/surveys/controllers/display.py b/indico/modules/events/surveys/controllers/display.py index 8bf695e781c..c8b91bfe4c6 100644 --- a/indico/modules/events/surveys/controllers/display.py +++ b/indico/modules/events/surveys/controllers/display.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, redirect, request, session from sqlalchemy.orm import joinedload from werkzeug.datastructures import MultiDict @@ -127,7 +125,7 @@ def _save_answers(self, form): self.submission = SurveySubmission(survey=survey, user=session.user) self.submission.is_anonymous = survey.anonymous for question in survey.questions: - answer = SurveyAnswer(question=question, data=getattr(form, 'question_{}'.format(question.id)).data) + answer = SurveyAnswer(question=question, data=getattr(form, f'question_{question.id}').data) self.submission.answers.append(answer) return self.submission @@ -139,7 +137,7 @@ def _check_access(self): raise Forbidden def _process(self): - pending_answers = {k: v for k, v in request.form.iterlists() if k.startswith('question_')} + pending_answers = {k: v for k, v in request.form.lists() if k.startswith('question_')} if not self.submission: self.submission = SurveySubmission(survey=self.survey, user=session.user) self.submission.pending_answers = pending_answers diff --git a/indico/modules/events/surveys/controllers/management/__init__.py b/indico/modules/events/surveys/controllers/management/__init__.py index 0cbc30cc485..ec94b424af0 100644 --- a/indico/modules/events/surveys/controllers/management/__init__.py +++ b/indico/modules/events/surveys/controllers/management/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import request from indico.modules.events.management.controllers import RHManageEventBase @@ -14,7 +12,7 @@ class RHManageSurveysBase(RHManageEventBase): - """Base class for all survey management RHs""" + """Base class for all survey management RHs.""" PERMISSION = 'surveys' @@ -30,4 +28,4 @@ class RHManageSurveyBase(RHManageSurveysBase): def _process_args(self): RHManageSurveysBase._process_args(self) - self.survey = Survey.find_one(id=request.view_args['survey_id'], is_deleted=False) + self.survey = Survey.query.filter_by(id=request.view_args['survey_id'], is_deleted=False).one() diff --git a/indico/modules/events/surveys/controllers/management/questionnaire.py b/indico/modules/events/surveys/controllers/management/questionnaire.py index 763501c7c21..fd920d4be15 100644 --- a/indico/modules/events/surveys/controllers/management/questionnaire.py +++ b/indico/modules/events/surveys/controllers/management/questionnaire.py @@ -1,16 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import json from flask import flash, jsonify, request, session -from sqlalchemy.orm import joinedload +from sqlalchemy.orm import contains_eager, joinedload from sqlalchemy.orm.exc import NoResultFound from werkzeug.datastructures import MultiDict from werkzeug.exceptions import NotFound @@ -33,7 +31,7 @@ class RHManageSurveyQuestionnaire(RHManageSurveyBase): - """Manage the questionnaire of a survey (question overview page)""" + """Manage the questionnaire of a survey (question overview page).""" def _process(self): field_types = get_field_types() @@ -43,7 +41,7 @@ def _process(self): class RHExportSurveyQuestionnaire(RHManageSurveyBase): - """Export the questionnaire to JSON format""" + """Export the questionnaire to JSON format.""" def _process(self): sections = [section.to_dict() for section in self.survey.sections] @@ -53,7 +51,7 @@ def _process(self): class RHImportSurveyQuestionnaire(RHManageSurveyBase): - """Import a questionnaire in JSON format""" + """Import a questionnaire in JSON format.""" def _remove_false_values(self, data): # The forms consider a missing key as False (and a False value as True) @@ -74,7 +72,7 @@ def _import_data(self, data): def _import_section(self, data): self._remove_false_values(data) - form = SectionForm(formdata=MultiDict(data.items()), csrf_enabled=False) + form = SectionForm(formdata=MultiDict(list(data.items())), csrf_enabled=False) if form.validate(): section = add_survey_section(self.survey, form.data) for item in data['content']: @@ -85,18 +83,18 @@ def _import_section(self, data): def _import_section_item(self, section, data): self._remove_false_values(data) if data['type'] == 'text': - form = TextForm(formdata=MultiDict(data.items()), csrf_enabled=False) + form = TextForm(formdata=MultiDict(list(data.items())), csrf_enabled=False) if form.validate(): add_survey_text(section, form.data) else: raise ValueError('Invalid text item') elif data['type'] == 'question': - for key, value in data['field_data'].iteritems(): + for key, value in data['field_data'].items(): if value is not None: data[key] = value field_cls = get_field_types()[data['field_type']] data = field_cls.process_imported_data(data) - form = field_cls.create_config_form(formdata=MultiDict(data.items()), csrf_enabled=False) + form = field_cls.create_config_form(formdata=MultiDict(list(data.items())), csrf_enabled=False) if not form.validate(): raise ValueError('Invalid question: {}'.format('\n'.join(form.error_list))) add_survey_question(section, field_cls, form.data) @@ -107,9 +105,9 @@ def _process(self): try: data = json.load(form.json_file.data.stream) self._import_data(data) - except ValueError as exception: + except ValueError as exc: db.session.rollback() - logger.info('%s tried to import an invalid JSON file: %s', session.user, exception.message) + logger.info('%s tried to import an invalid JSON file: %s', session.user, exc) flash(_("Invalid file selected."), 'error') else: flash(_("The questionnaire has been imported."), 'success') @@ -119,7 +117,7 @@ def _process(self): class RHManageSurveySectionBase(RHManageSurveysBase): - """Base class for RHs that deal with a specific survey section""" + """Base class for RHs that deal with a specific survey section.""" normalize_url_spec = { 'locators': { @@ -130,13 +128,17 @@ class RHManageSurveySectionBase(RHManageSurveysBase): def _process_args(self): RHManageSurveysBase._process_args(self) - self.section = SurveySection.find_one(SurveySection.id == request.view_args['section_id'], ~Survey.is_deleted, - _join=SurveySection.survey, _eager=SurveySection.survey) + self.section = (SurveySection.query + .filter(SurveySection.id == request.view_args['section_id'], + ~Survey.is_deleted) + .join(SurveySection.survey) + .options(contains_eager(SurveySection.survey)) + .one()) self.survey = self.section.survey class RHManageSurveyTextBase(RHManageSurveysBase): - """Base class for RHs that deal with a specific survey text item""" + """Base class for RHs that deal with a specific survey text item.""" normalize_url_spec = { 'locators': { @@ -146,13 +148,17 @@ class RHManageSurveyTextBase(RHManageSurveysBase): def _process_args(self): RHManageSurveysBase._process_args(self) - self.text = SurveyText.find_one(SurveyText.id == request.view_args['text_id'], ~Survey.is_deleted, - _join=SurveyText.survey, _eager=SurveyText.survey) + self.text = (SurveyText.query + .filter(SurveyText.id == request.view_args['text_id'], + ~Survey.is_deleted) + .join(SurveyText.survey) + .options(contains_eager(SurveyText.survey)) + .one()) self.survey = self.text.survey class RHManageSurveyQuestionBase(RHManageSurveysBase): - """Base class for RHs that deal with a specific survey question""" + """Base class for RHs that deal with a specific survey question.""" normalize_url_spec = { 'locators': { @@ -162,14 +168,17 @@ class RHManageSurveyQuestionBase(RHManageSurveysBase): def _process_args(self): RHManageSurveysBase._process_args(self) - self.question = SurveyQuestion.find_one(SurveyQuestion.id == request.view_args['question_id'], - ~Survey.is_deleted, - _join=SurveyQuestion.survey, _eager=SurveyQuestion.survey) + self.question = (SurveyQuestion.query + .filter(SurveyQuestion.id == request.view_args['question_id'], + ~Survey.is_deleted) + .join(SurveyQuestion.survey) + .options(contains_eager(SurveyQuestion.survey)) + .one()) self.survey = self.question.survey class RHAddSurveySection(RHManageSurveyBase): - """Add a new section to a survey""" + """Add a new section to a survey.""" def _process(self): form = SectionForm() @@ -185,7 +194,7 @@ def _process(self): class RHEditSurveySection(RHManageSurveySectionBase): - """Edit a survey section""" + """Edit a survey section.""" def _process(self): form = SectionForm(obj=FormDefaults(self.section)) @@ -204,7 +213,7 @@ def _process(self): class RHDeleteSurveySection(RHManageSurveySectionBase): - """Delete a survey section and all its questions""" + """Delete a survey section and all its questions.""" def _process(self): db.session.delete(self.section) @@ -219,7 +228,7 @@ def _process(self): class RHAddSurveyText(RHManageSurveySectionBase): - """Add a new text item to a survey""" + """Add a new text item to a survey.""" def _process(self): form = TextForm() @@ -231,7 +240,7 @@ def _process(self): class RHEditSurveyText(RHManageSurveyTextBase): - """Edit a survey text item""" + """Edit a survey text item.""" def _process(self): form = TextForm(obj=FormDefaults(self.text)) @@ -245,7 +254,7 @@ def _process(self): class RHDeleteSurveyText(RHManageSurveyTextBase): - """Delete a survey text item""" + """Delete a survey text item.""" def _process(self): db.session.delete(self.text) @@ -256,7 +265,7 @@ def _process(self): class RHAddSurveyQuestion(RHManageSurveySectionBase): - """Add a new question to a survey""" + """Add a new question to a survey.""" def _process(self): try: @@ -285,7 +294,7 @@ def _process(self): class RHEditSurveyQuestion(RHManageSurveyQuestionBase): - """Edit a survey question""" + """Edit a survey question.""" def _process(self): form = self.question.field.create_config_form(obj=FormDefaults(self.question, **self.question.field_data)) @@ -300,7 +309,7 @@ def _process(self): class RHDeleteSurveyQuestion(RHManageSurveyQuestionBase): - """Delete a survey question""" + """Delete a survey question.""" def _process(self): db.session.delete(self.question) @@ -311,28 +320,34 @@ def _process(self): class RHSortSurveyItems(RHManageSurveyBase): - """Sort survey items/sections and/or move items them between sections""" + """Sort survey items/sections and/or move items them between sections.""" def _sort_sections(self): sections = {section.id: section for section in self.survey.sections} - section_ids = map(int, request.form.getlist('section_ids')) + section_ids = request.form.getlist('section_ids', type=int) for position, section_id in enumerate(section_ids, 1): sections[section_id].position = position logger.info('Sections in %s reordered by %s', self.survey, session.user) def _sort_items(self): - section = SurveySection.find_one(survey=self.survey, id=request.form['section_id'], - _eager=SurveySection.children) + section = (SurveySection.query + .filter_by(survey=self.survey, id=request.form['section_id']) + .options(joinedload(SurveySection.children)) + .one()) section_items = {x.id: x for x in section.children} - item_ids = map(int, request.form.getlist('item_ids')) + item_ids = request.form.getlist('item_ids', type=int) changed_section = None for position, item_id in enumerate(item_ids, 1): try: section_items[item_id].position = position except KeyError: # item is not in section, was probably moved - item = SurveyItem.find_one(SurveyItem.survey == self.survey, SurveyItem.id == item_id, - SurveyItem.type != SurveyItemType.section, _eager=SurveyItem.parent) + item = (SurveyItem.query + .filter(SurveyItem.survey == self.survey, + SurveyItem.id == item_id, + SurveyItem.type != SurveyItemType.section) + .options(joinedload(SurveyItem.parent)) + .one()) changed_section = item.parent item.position = position item.parent = section @@ -354,8 +369,8 @@ def _process(self): def _render_questionnaire_preview(survey): # load the survey once again with all the necessary data - survey = (Survey - .find(id=survey.id) + survey = (Survey.query + .filter_by(id=survey.id) .options(joinedload(Survey.sections).joinedload(SurveySection.children)) .one()) tpl = get_template_module('events/surveys/management/_questionnaire_preview.html') diff --git a/indico/modules/events/surveys/controllers/management/results.py b/indico/modules/events/surveys/controllers/management/results.py index 778f93d692d..35ad5d524d7 100644 --- a/indico/modules/events/surveys/controllers/management/results.py +++ b/indico/modules/events/surveys/controllers/management/results.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, jsonify, redirect, request from sqlalchemy.orm import defaultload, joinedload @@ -24,7 +22,7 @@ class RHSurveyResults(RHManageSurveyBase): - """Displays summarized results of the survey""" + """Display summarized results of the survey.""" def _process_args(self): RHManageSurveysBase._process_args(self) @@ -39,7 +37,7 @@ def _process(self): class RHExportSubmissionsBase(RHManageSurveyBase): - """Export submissions from the survey""" + """Export submissions from the survey.""" CSRF_ENABLED = False ALLOW_LOCKED = True @@ -51,7 +49,7 @@ def _process(self): submission_ids = set(map(int, request.form.getlist('submission_ids'))) headers, rows = generate_spreadsheet_from_survey(self.survey, submission_ids) - filename = 'submissions-{}'.format(self.survey.id) + filename = f'submissions-{self.survey.id}' return self._export(filename, headers, rows) def _export(self, filename, headers, rows): @@ -59,14 +57,14 @@ def _export(self, filename, headers, rows): class RHExportSubmissionsCSV(RHExportSubmissionsBase): - """Export submissions as CSV""" + """Export submissions as CSV.""" def _export(self, filename, headers, rows): return send_csv(filename + '.csv', headers, rows) class RHExportSubmissionsExcel(RHExportSubmissionsBase): - """Export submissions as XLSX""" + """Export submissions as XLSX.""" def _export(self, filename, headers, rows): return send_xlsx(filename + '.xlsx', headers, rows, tz=self.event.tzinfo) @@ -84,14 +82,14 @@ def _process_args(self): survey_strategy = joinedload('survey') answers_strategy = defaultload('answers').joinedload('question') sections_strategy = joinedload('survey').defaultload('sections').joinedload('children') - self.submission = (SurveySubmission - .find(id=request.view_args['submission_id']) + self.submission = (SurveySubmission.query + .filter_by(id=request.view_args['submission_id']) .options(answers_strategy, survey_strategy, sections_strategy) .one()) class RHDeleteSubmissions(RHManageSurveyBase): - """Remove submissions from the survey""" + """Remove submissions from the survey.""" def _process(self): submission_ids = set(map(int, request.form.getlist('submission_ids'))) @@ -100,14 +98,14 @@ def _process(self): self.survey.submissions.remove(submission) logger.info('Submission %s deleted from survey %s', submission, self.survey) self.event.log(EventLogRealm.management, EventLogKind.negative, 'Surveys', - 'Submission removed from survey "{}"'.format(self.survey.title), + f'Submission removed from survey "{self.survey.title}"', data={'Submitter': submission.user.full_name if not submission.is_anonymous else 'Anonymous'}) return jsonify(success=True) class RHDisplaySubmission(RHSurveySubmissionBase): - """Display a single submission-page""" + """Display a single submission-page.""" def _process(self): answers = {answer.question_id: answer for answer in self.submission.answers} diff --git a/indico/modules/events/surveys/controllers/management/survey.py b/indico/modules/events/surveys/controllers/management/survey.py index 78ea3bbd0c9..64d63529285 100644 --- a/indico/modules/events/surveys/controllers/management/survey.py +++ b/indico/modules/events/surveys/controllers/management/survey.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import flash, redirect, session from indico.core.db import db @@ -26,7 +24,7 @@ class RHManageSurveys(RHManageSurveysBase): - """Survey management overview (list of surveys)""" + """Survey management overview (list of surveys).""" def _process(self): surveys = (Survey.query.with_parent(self.event) @@ -37,7 +35,7 @@ def _process(self): class RHManageSurvey(RHManageSurveyBase): - """Specific survey management (overview)""" + """Specific survey management (overview).""" def _process(self): submitted_surveys = [s for s in self.survey.submissions if s.is_submitted] @@ -46,7 +44,7 @@ def _process(self): class RHEditSurvey(RHManageSurveyBase): - """Edit a survey's basic data/settings""" + """Edit a survey's basic data/settings.""" def _get_form_defaults(self): return FormDefaults(self.survey, limit_submissions=self.survey.submission_limit is not None) @@ -64,7 +62,7 @@ def _process(self): class RHDeleteSurvey(RHManageSurveyBase): - """Delete a survey""" + """Delete a survey.""" def _process(self): self.survey.is_deleted = True @@ -74,7 +72,7 @@ def _process(self): class RHCreateSurvey(RHManageSurveysBase): - """Create a new survey""" + """Create a new survey.""" def _process(self): form = SurveyForm(obj=FormDefaults(require_user=True), event=self.event) @@ -93,7 +91,7 @@ def _process(self): class RHScheduleSurvey(RHManageSurveyBase): - """Schedule a survey's start/end dates""" + """Schedule a survey's start/end dates.""" def _get_form_defaults(self): return FormDefaults(self.survey) @@ -117,7 +115,7 @@ def _process(self): class RHCloseSurvey(RHManageSurveyBase): - """Close a survey (prevent users from submitting responses)""" + """Close a survey (prevent users from submitting responses).""" def _process(self): self.survey.close() @@ -127,7 +125,7 @@ def _process(self): class RHOpenSurvey(RHManageSurveyBase): - """Open a survey (allows users to submit responses)""" + """Open a survey (allows users to submit responses).""" def _process(self): if self.survey.state == SurveyState.finished: @@ -142,7 +140,7 @@ def _process(self): class RHSendSurveyLinks(RHManageSurveyBase): - """Send emails with URL of the survey""" + """Send emails with URL of the survey.""" def _process(self): tpl = get_template_module('events/surveys/emails/survey_link_email.html', event=self.event) diff --git a/indico/modules/events/surveys/fields/__init__.py b/indico/modules/events/surveys/fields/__init__.py index f4332da4d1d..542b60b9dcf 100644 --- a/indico/modules/events/surveys/fields/__init__.py +++ b/indico/modules/events/surveys/fields/__init__.py @@ -1,26 +1,24 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core import signals from indico.modules.events.surveys.fields.base import SurveyField from indico.web.fields import get_field_definitions def get_field_types(): - """Gets a dict containing all field types""" + """Get a dict containing all field types.""" return get_field_definitions(SurveyField) @signals.get_fields.connect_via(SurveyField) def _get_fields(sender, **kwargs): - from .simple import SurveyTextField, SurveyNumberField, SurveyBoolField - from .choices import SurveySingleChoiceField, SurveyMultiSelectField + from .choices import SurveyMultiSelectField, SurveySingleChoiceField + from .simple import SurveyBoolField, SurveyNumberField, SurveyTextField yield SurveyTextField yield SurveyNumberField yield SurveyBoolField diff --git a/indico/modules/events/surveys/fields/base.py b/indico/modules/events/surveys/fields/base.py index 93b6e6fcc78..a34a5861672 100644 --- a/indico/modules/events/surveys/fields/base.py +++ b/indico/modules/events/surveys/fields/base.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from wtforms.fields import BooleanField, StringField, TextAreaField from wtforms.validators import DataRequired diff --git a/indico/modules/events/surveys/fields/choices.py b/indico/modules/events/surveys/fields/choices.py index c5cff630e88..ef12229acec 100644 --- a/indico/modules/events/surveys/fields/choices.py +++ b/indico/modules/events/surveys/fields/choices.py @@ -1,14 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import division, unicode_literals - import uuid -from collections import Counter, OrderedDict +from collections import Counter from copy import deepcopy from indico.modules.events.surveys.fields.base import SurveyField @@ -17,14 +15,14 @@ from indico.web.fields.choices import MultiSelectField, SingleChoiceField -class _AddUUIDMixin(object): +class _AddUUIDMixin: @staticmethod def process_imported_data(data): - """Generate the options' IDs""" + """Generate the options' IDs.""" data = deepcopy(data) if 'options' in data: for option in data['options']: - option['id'] = unicode(uuid.uuid4()) + option['id'] = str(uuid.uuid4()) return data @@ -39,9 +37,9 @@ def get_summary(self): no_option = {'id': None, 'option': _("No selection")} options.append(no_option) return {'total': total, - 'labels': [alpha_enum(val).upper() for val in xrange(len(options))], - 'absolute': OrderedDict((opt['option'], counter[opt['id']]) for opt in options), - 'relative': OrderedDict((opt['option'], counter[opt['id']] / total) for opt in options)} + 'labels': [alpha_enum(val).upper() for val in range(len(options))], + 'absolute': {opt['option']: counter[opt['id']] for opt in options}, + 'relative': {opt['option']: counter[opt['id']] / total for opt in options}} class SurveyMultiSelectField(_AddUUIDMixin, MultiSelectField, SurveyField): @@ -52,6 +50,6 @@ def get_summary(self): total = sum(counter.values()) options = self.object.field_data['options'] return {'total': total, - 'labels': [alpha_enum(val).upper() for val in xrange(len(options))], - 'absolute': OrderedDict((opt['option'], counter[opt['id']]) for opt in options), - 'relative': OrderedDict((opt['option'], counter[opt['id']] / total if total else 0) for opt in options)} + 'labels': [alpha_enum(val).upper() for val in range(len(options))], + 'absolute': {opt['option']: counter[opt['id']] for opt in options}, + 'relative': {opt['option']: counter[opt['id']] / total if total else 0 for opt in options}} diff --git a/indico/modules/events/surveys/fields/simple.py b/indico/modules/events/surveys/fields/simple.py index b8a834f91a2..c9a58877e3c 100644 --- a/indico/modules/events/surveys/fields/simple.py +++ b/indico/modules/events/surveys/fields/simple.py @@ -1,13 +1,11 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import division, unicode_literals - -from collections import Counter, OrderedDict +from collections import Counter from indico.modules.events.surveys.fields.base import SurveyField from indico.util.i18n import _ @@ -32,8 +30,8 @@ def get_summary(self): results = {'total': sum(counter.elements()), 'max': max(counter.elements()), 'min': min(counter.elements()), - 'absolute': OrderedDict(sorted(counter.iteritems())), - 'relative': OrderedDict((k, v / total_answers) for k, v in sorted(counter.iteritems()))} + 'absolute': dict(sorted(counter.items())), + 'relative': {k: v / total_answers for k, v in sorted(counter.items())}} results['average'] = results['total'] / len(list(counter.elements())) return results @@ -47,5 +45,5 @@ def get_summary(self): return total = sum(counter.values()) return {'total': total, - 'absolute': OrderedDict(((_('Yes'), counter[True]), (_('No'), counter[False]))), - 'relative': OrderedDict(((_('Yes'), counter[True] / total), (_('No'), counter[False] / total)))} + 'absolute': {_('Yes'): counter[True], _('No'): counter[False]}, + 'relative': {_('Yes'): counter[True] / total, _('No'): counter[False] / total}} diff --git a/indico/modules/events/surveys/forms.py b/indico/modules/events/surveys/forms.py index 72e16bf0dd5..34277165c20 100644 --- a/indico/modules/events/surveys/forms.py +++ b/indico/modules/events/surveys/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import time from flask import request @@ -59,7 +57,7 @@ class SurveyForm(IndicoForm): def __init__(self, *args, **kwargs): self.event = kwargs.pop('event') - super(SurveyForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def validate_title(self, field): query = (Survey.query.with_parent(self.event) @@ -67,7 +65,7 @@ def validate_title(self, field): Survey.title != field.object_data, ~Survey.is_deleted)) if query.count(): - raise ValidationError(_('There is already a survey named "{}" on this event'.format(escape(field.data)))) + raise ValidationError(_(f'There is already a survey named "{escape(field.data)}" on this event')) def post_validate(self): if not self.anonymous.data: @@ -88,7 +86,7 @@ def __init__(self, *args, **kwargs): survey = kwargs.pop('survey') self.allow_reschedule_start = kwargs.pop('allow_reschedule_start') self.timezone = survey.event.timezone - super(ScheduleSurveyForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) if not survey.start_notification_sent or not self.allow_reschedule_start: del self.resend_start_notification @@ -123,12 +121,12 @@ class InvitationForm(IndicoForm): def __init__(self, *args, **kwargs): event = kwargs.pop('event') - super(InvitationForm, self).__init__(*args, **kwargs) - self.from_address.choices = event.get_allowed_sender_emails().items() + super().__init__(*args, **kwargs) + self.from_address.choices = list(event.get_allowed_sender_emails().items()) self.body.description = render_placeholder_info('survey-link-email', event=None, survey=None) def is_submitted(self): - return super(InvitationForm, self).is_submitted() and 'submitted' in request.form + return super().is_submitted() and 'submitted' in request.form def validate_body(self, field): missing = get_missing_placeholders('survey-link-email', field.data, event=None, survey=None) diff --git a/indico/modules/events/surveys/models/items.py b/indico/modules/events/surveys/models/items.py index db016135c5a..21342c2eff1 100644 --- a/indico/modules/events/surveys/models/items.py +++ b/indico/modules/events/surveys/models/items.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import JSONB from sqlalchemy.event import listens_for @@ -14,15 +12,17 @@ from indico.core.db.sqlalchemy import PyIntEnum from indico.core.db.sqlalchemy.descriptions import DescriptionMixin, RenderMode from indico.modules.events.surveys.fields import get_field_types -from indico.util.string import return_ascii, text_to_repr -from indico.util.struct.enum import IndicoEnum +from indico.util.enum import IndicoEnum +from indico.util.string import text_to_repr def _get_next_position(context): """Get the next question position for the event.""" survey_id = context.current_parameters['survey_id'] parent_id = context.current_parameters['parent_id'] - res = db.session.query(db.func.max(SurveyItem.position)).filter_by(survey_id=survey_id, parent_id=parent_id).one() + res = (db.session.query(db.func.max(SurveyItem.position)) + .filter(SurveyItem.survey_id == survey_id, SurveyItem.parent_id == parent_id) + .one()) return (res[0] or 0) + 1 @@ -171,16 +171,15 @@ def not_empty_answers(self): return [a for a in self.answers if not a.is_empty] def get_summary(self, **kwargs): - """Returns the summary of answers submitted for this question.""" + """Return the summary of answers submitted for this question.""" if self.field: return self.field.get_summary(**kwargs) - @return_ascii def __repr__(self): - return ''.format(self.id, self.survey_id, self.field_type, self.title) + return f'' def to_dict(self): - data = super(SurveyQuestion, self).to_dict() + data = super().to_dict() data.update({'is_required': self.is_required, 'field_type': self.field_type, 'field_data': self.field.copy_field_data()}) return data @@ -206,12 +205,11 @@ class SurveySection(SurveyItem): def locator(self): return dict(self.survey.locator, section_id=self.id) - @return_ascii def __repr__(self): - return ''.format(self.id, self.survey_id, self.title) + return f'' def to_dict(self): - data = super(SurveySection, self).to_dict() + data = super().to_dict() content = [child.to_dict() for child in self.children] data.update({'content': content, 'display_as_section': self.display_as_section}) if not self.display_as_section: @@ -229,13 +227,12 @@ class SurveyText(SurveyItem): def locator(self): return dict(self.survey.locator, section_id=self.parent_id, text_id=self.id) - @return_ascii def __repr__(self): desc = text_to_repr(self.description) - return ''.format(self.id, self.survey_id, desc) + return f'' def to_dict(self): - data = super(SurveyText, self).to_dict() + data = super().to_dict() del data['title'] return data diff --git a/indico/modules/events/surveys/models/submissions.py b/indico/modules/events/surveys/models/submissions.py index 24a5d5b3b8a..e42774937c6 100644 --- a/indico/modules/events/surveys/models/submissions.py +++ b/indico/modules/events/surveys/models/submissions.py @@ -1,18 +1,15 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.dialects.postgresql import JSONB from indico.core.db import db from indico.core.db.sqlalchemy import UTCDateTime from indico.core.db.sqlalchemy.util.queries import increment_and_get -from indico.util.string import return_ascii def _get_next_friendly_id(context): @@ -107,9 +104,8 @@ class SurveySubmission(db.Model): def locator(self): return dict(self.survey.locator, submission_id=self.id) - @return_ascii def __repr__(self): - return ''.format(self.id, self.survey_id, self.user_id) + return f'' class SurveyAnswer(db.Model): @@ -152,9 +148,8 @@ class SurveyAnswer(db.Model): def is_empty(self): return self.question.field.is_value_empty(self) - @return_ascii def __repr__(self): - return ''.format(self.submission_id, self.question_id, self.data) + return f'' @property def answer_data(self): diff --git a/indico/modules/events/surveys/models/surveys.py b/indico/modules/events/surveys/models/surveys.py index 00a05ae37d1..0f18de9babf 100644 --- a/indico/modules/events/surveys/models/surveys.py +++ b/indico/modules/events/surveys/models/surveys.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from uuid import uuid4 from sqlalchemy import inspect @@ -22,9 +20,8 @@ from indico.modules.events.registration.models.registrations import Registration from indico.modules.events.surveys import logger from indico.util.date_time import now_utc +from indico.util.enum import IndicoEnum from indico.util.locators import locator_property -from indico.util.string import return_ascii -from indico.util.struct.enum import IndicoEnum from indico.web.flask.templating import get_template_module @@ -63,7 +60,7 @@ class Survey(db.Model): UUID, unique=True, nullable=False, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) # An introduction text for users of the survey introduction = db.Column( @@ -216,12 +213,12 @@ def has_started(cls): @locator_property def locator(self): - return {'confId': self.event_id, + return {'event_id': self.event_id, 'survey_id': self.id} @locator.token def locator(self): - """A locator that adds the UUID if the survey is private""" + """A locator that adds the UUID if the survey is private.""" token = self.uuid if self.private else None return dict(self.locator, token=token) @@ -243,7 +240,7 @@ def limit_reached(self): @property def start_notification_recipients(self): - """Returns all recipients of the notifications. + """Return all recipients of the notifications. This includes both explicit recipients and, if enabled, participants of the event. @@ -263,7 +260,7 @@ def is_active(cls): submissions = (db.session.query(db.func.count(db.m.SurveySubmission.id)) .filter(db.m.SurveySubmission.survey_id == cls.id) .correlate(Survey) - .as_scalar()) + .scalar_subquery()) limit_criterion = db.case([(cls.submission_limit.is_(None), True)], else_=(submissions < cls.submission_limit)) return ~cls.is_deleted & cls.questions.any() & cls.has_started & ~cls.has_ended & limit_criterion @@ -278,9 +275,8 @@ def is_visible(self): def is_visible(cls): return ~cls.is_deleted & cls.questions.any() & cls.has_started - @return_ascii def __repr__(self): - return ''.format(self.id, self.event_id, self.title) + return f'' def can_submit(self, user): return self.is_active and (not self.require_user or user) diff --git a/indico/modules/events/surveys/operations.py b/indico/modules/events/surveys/operations.py index b945b177e88..14596c9317e 100644 --- a/indico/modules/events/surveys/operations.py +++ b/indico/modules/events/surveys/operations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core.db import db diff --git a/indico/modules/events/surveys/placeholders.py b/indico/modules/events/surveys/placeholders.py index 160074a5beb..18b53a0ff43 100644 --- a/indico/modules/events/surveys/placeholders.py +++ b/indico/modules/events/surveys/placeholders.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from markupsafe import Markup, escape from indico.util.i18n import _ diff --git a/indico/modules/events/surveys/tasks.py b/indico/modules/events/surveys/tasks.py index 47ed535a1e4..cce3c2bbeee 100644 --- a/indico/modules/events/surveys/tasks.py +++ b/indico/modules/events/surveys/tasks.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from celery.schedules import crontab from indico.core.celery import celery @@ -16,7 +14,11 @@ @celery.periodic_task(name='survey_start_notifications', run_every=crontab(minute='*/30')) def send_start_notifications(): - active_surveys = Survey.find_all(Survey.is_active, ~Survey.start_notification_sent, Survey.notifications_enabled) + active_surveys = (Survey.query + .filter(Survey.is_active, + ~Survey.start_notification_sent, + Survey.notifications_enabled) + .all()) for survey in active_surveys: survey.send_start_notification() db.session.commit() diff --git a/indico/modules/events/surveys/templates/management/survey_results.html b/indico/modules/events/surveys/templates/management/survey_results.html index 5511fb8c501..41162f6e08f 100644 --- a/indico/modules/events/surveys/templates/management/survey_results.html +++ b/indico/modules/events/surveys/templates/management/survey_results.html @@ -96,7 +96,7 @@

    {{ section.title }}

    {% macro _choice_answer(question, chart_type, short_labels=false) %} {% set summary = question.get_summary() %}
    - {% for label, value in summary.relative.iteritems() %} + {% for label, value in summary.relative.items() %}
    {{ loop.index0|alpha_enum|upper }}. {{ label }}: {{ summary.absolute[label] }} ({{ '%0.2f'|format(value * 100) }}%)
    @@ -113,7 +113,7 @@

    {{ section.title }}

    {% trans %}# of times chosen{% endtrans %}
    {% endif %}
    diff --git a/indico/modules/events/surveys/util.py b/indico/modules/events/surveys/util.py index 8af23a2111e..8be5315cd69 100644 --- a/indico/modules/events/surveys/util.py +++ b/indico/modules/events/surveys/util.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from operator import attrgetter from flask import session @@ -21,34 +19,34 @@ def make_survey_form(survey): - """Creates a WTForm from survey questions. + """Create a WTForm from survey questions. Each question will use a field named ``question_ID``. :param survey: The `Survey` for which to create the form. :return: An `IndicoForm` subclass. """ - form_class = type(b'SurveyForm', (IndicoForm,), {}) + form_class = type('SurveyForm', (IndicoForm,), {}) for question in survey.questions: field_impl = question.field if field_impl is None: # field definition is not available anymore continue - name = 'question_{}'.format(question.id) + name = f'question_{question.id}' setattr(form_class, name, field_impl.create_wtf_field()) return form_class def save_submitted_survey_to_session(submission): - """Save submission of a survey to session for further checks""" + """Save submission of a survey to session for further checks.""" session.setdefault('submitted_surveys', {})[submission.survey.id] = submission.id session.modified = True @memoize_request def was_survey_submitted(survey): - """Check whether the current user has submitted a survey""" + """Check whether the current user has submitted a survey.""" from indico.modules.events.surveys.models.surveys import Survey query = (Survey.query.with_parent(survey.event) .filter(Survey.submissions.any(db.and_(SurveySubmission.is_submitted, @@ -59,11 +57,11 @@ def was_survey_submitted(survey): submission_id = session.get('submitted_surveys', {}).get(survey.id) if submission_id is None: return False - return SurveySubmission.find(id=submission_id, is_submitted=True).has_rows() + return SurveySubmission.query.filter_by(id=submission_id, is_submitted=True).has_rows() def is_submission_in_progress(survey): - """Check whether the current user has a survey submission in progress""" + """Check whether the current user has a survey submission in progress.""" from indico.modules.events.surveys.models.surveys import Survey if session.user: query = (Survey.query.with_parent(survey.event) @@ -76,7 +74,7 @@ def is_submission_in_progress(survey): def generate_spreadsheet_from_survey(survey, submission_ids): - """Generates spreadsheet data from a given survey. + """Generate spreadsheet data from a given survey. :param survey: `Survey` for which the user wants to export submissions :param submission_ids: The list of submissions to include in the file @@ -104,25 +102,29 @@ def generate_spreadsheet_from_survey(survey, submission_ids): def _format_title(question): if question.parent.title: - return '{}: {}'.format(question.parent.title, question.title) + return f'{question.parent.title}: {question.title}' else: return question.title def _filter_submissions(survey, submission_ids): if submission_ids: - return SurveySubmission.find_all(SurveySubmission.id.in_(submission_ids), survey=survey) + return (SurveySubmission.query + .filter(SurveySubmission.id.in_(submission_ids), + SurveySubmission.survey == survey) + .all()) return [x for x in survey.submissions if x.is_submitted] def get_events_with_submitted_surveys(user, dt=None): - """Gets the IDs of events where the user submitted a survey. + """Get the IDs of events where the user submitted a survey. :param user: A `User` :param dt: Only include events taking place on/after that date :return: A set of event ids """ from indico.modules.events.surveys.models.surveys import Survey + # Survey submissions are not stored in links anymore, so we need to get them directly query = (user.survey_submissions .options(load_only('survey_id')) diff --git a/indico/modules/events/surveys/views.py b/indico/modules/events/surveys/views.py index 725e0d47bf8..e032bcd198d 100644 --- a/indico/modules/events/surveys/views.py +++ b/indico/modules/events/surveys/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.management.views import WPEventManagement from indico.modules.events.views import WPConferenceDisplayBase, WPSimpleEventDisplayBase from indico.web.views import WPJinjaMixin diff --git a/indico/modules/events/templates/display/common/_legacy.html b/indico/modules/events/templates/display/common/_legacy.html index 31fdec05e49..ba179048f1b 100644 --- a/indico/modules/events/templates/display/common/_legacy.html +++ b/indico/modules/events/templates/display/common/_legacy.html @@ -9,8 +9,8 @@ start_time=event.start_dt|format_time(format='medium', timezone=timezone), end_date=event.end_dt|format_date(format='full', timezone=timezone), end_time=event.end_dt|format_time(format='medium', timezone=timezone) -%} - from {{start_date}} ({{start_time}}) - to {{end_date}} ({{end_time}}) + from {{ start_date }} ({{ start_time }}) + to {{ end_date }} ({{ end_time }}) {%- endtrans %} {% endif %} {% endmacro %} @@ -19,7 +19,7 @@ {#- -#} - {%- if item.venue_name and (not parent or parent.venue_name != item.venue_name) -%} + {%- if item.venue_name -%} {{ item.venue_name }} {%- if item.room_name %} ({{ item.get_room_name(full=false) }}) diff --git a/indico/modules/events/templates/display/conference/base.html b/indico/modules/events/templates/display/conference/base.html index d2ce5e3e7d3..1dea50086e5 100644 --- a/indico/modules/events/templates/display/conference/base.html +++ b/indico/modules/events/templates/display/conference/base.html @@ -55,12 +55,15 @@

    {% include 'flashed_messages.html' %} {{ render_event_header_msg(event, meeting=false) }} + {% set visible_menu_entries = conf_layout_params.menu | selectattr('is_visible') | list %}
    -
      - {%- for entry in conf_layout_params.menu if entry.is_visible %} - {{ menu_entry_display(entry, active_entry_id=conf_layout_params.active_menu_item) }} - {% endfor -%} -
    + {% if visible_menu_entries %} +
      + {%- for entry in visible_menu_entries %} + {{ menu_entry_display(entry, active_entry_id=conf_layout_params.active_menu_item) }} + {% endfor -%} +
    + {% endif %} {% if event.contact_emails or event.contact_phones -%}
    diff --git a/indico/modules/events/templates/display/event_ical_export.html b/indico/modules/events/templates/display/event_ical_export.html deleted file mode 100644 index 4fc8d18bbb4..00000000000 --- a/indico/modules/events/templates/display/event_ical_export.html +++ /dev/null @@ -1,38 +0,0 @@ -{% extends '_ical_export.html' %} - -{% block download_text %} - {% trans %}Download current event:{% endtrans %} -{% endblock %} - -{% block extra_download %} - {% if item.type_.name != 'lecture' %} -
    - - {% trans %}Detailed timetable calendar file{% endtrans %} - -
    - {% endif %} -{% endblock %} - -{% block extra_info %} - {% if item.type_.name != 'lecture' %} - {% trans %}Detailed timetable{% endtrans %} - {% endif %} -{% endblock %} - -{% block javascript %} - {{ super() }} - - -{% endblock %} diff --git a/indico/modules/events/templates/display/indico/_common.html b/indico/modules/events/templates/display/indico/_common.html index fabb3bc39e1..f7f1b5274e7 100644 --- a/indico/modules/events/templates/display/indico/_common.html +++ b/indico/modules/events/templates/display/indico/_common.html @@ -58,22 +58,22 @@

    {{ item.venue_name }}

    {{ user_data.display_full_name }} {%- if user_data.affiliation -%} - ({#--#} + ({#--#} {%- if italic_affiliation -%} {{ user_data.affiliation }} {%- else -%} {{ user_data.affiliation }} {%- endif -%} + {#--#}){#--#} - {#--#}){#--#} {%- endif -%} {%- endmacro %} {% macro render_users(user_list, span_class='', title=true, italic_affiliation=false, separator=', ') -%} {%- for link in user_list -%} - {%- if caller -%} + {%- if caller is defined -%} {{- caller(link) -}} {%- else -%} diff --git a/indico/modules/events/templates/footer.html b/indico/modules/events/templates/footer.html index fd0027fb8cf..6634e9b2143 100644 --- a/indico/modules/events/templates/footer.html +++ b/indico/modules/events/templates/footer.html @@ -45,7 +45,7 @@

    {% trans %}Calendaring{% endtrans %}

    diff --git a/indico/modules/events/templates/header.html b/indico/modules/events/templates/header.html index e23df777d7c..1711853ce97 100644 --- a/indico/modules/events/templates/header.html +++ b/indico/modules/events/templates/header.html @@ -10,12 +10,12 @@ {% if rel.first is not none %} - {% endif %} {% if rel.prev is not none %} - {% endif %} @@ -23,12 +23,12 @@ title="{% trans %}Up to category{% endtrans %}"> {% if rel.next is not none %} - {% endif %} {% if rel.last is not none %} - {% endif %} @@ -175,7 +175,8 @@ {%- set dates = event.iter_days(tzinfo=event.tzinfo)|list -%} -{%- set show_filter_button = event.type == 'meeting' and (dates|length > 1 or event.sessions) -%} +{%- set show_filter_button = (event.type == 'meeting' or (event.type == 'conference' and theme)) + and (dates|length > 1 or event.sessions) -%} {%- set filters_active = show_filter_button and (request.args.get('showDate', 'all') != 'all' or request.args.get('showSession', 'all') != 'all' or request.args.get('detailLevel', 'contribution') != 'contribution') -%} @@ -200,9 +201,8 @@ {{ _render_filters() }} {% endif %} - - {{ template_hook('event-ical-export', event=event) }} + {% if event.type == 'meeting' %} - $(document).ready(function() { - $('.js-export-ical').on('click', function(evt) { - evt.preventDefault(); - $(this).trigger('menu_select'); - }); - }); - diff --git a/indico/modules/events/templates/management/_lists.html b/indico/modules/events/templates/management/_lists.html index c8693fd137c..71ffb17e8a9 100644 --- a/indico/modules/events/templates/management/_lists.html +++ b/indico/modules/events/templates/management/_lists.html @@ -8,7 +8,7 @@ {% macro render_filter_statistics(displayed_num, total_num, total_duration) %} {{ render_displayed_entries_fragment(displayed_num, total_num) }} + title="{% trans total=total_duration[0]|format_human_timedelta, scheduled=total_duration[1]|format_human_timedelta %}{{ total }} total duration of which {{ scheduled }} are scheduled{% endtrans %}"> {{ total_duration[0] | format_human_timedelta }} {% endmacro %} diff --git a/indico/modules/events/templates/reviewing_questions_management.html b/indico/modules/events/templates/reviewing_questions_management.html index e2a4a36651f..d3b94e23715 100644 --- a/indico/modules/events/templates/reviewing_questions_management.html +++ b/indico/modules/events/templates/reviewing_questions_management.html @@ -7,7 +7,7 @@ {%- trans %}Add new question{% endtrans -%}
      - {% for name, field_type in field_types.iteritems() -%} + {% for name, field_type in field_types.items() -%}
    • - {{- user.name[0] -}} +
      +
      {% endmacro %} diff --git a/indico/modules/events/timetable/__init__.py b/indico/modules/events/timetable/__init__.py index 35e74350d75..bead2494d88 100644 --- a/indico/modules/events/timetable/__init__.py +++ b/indico/modules/events/timetable/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import render_template, session from indico.core import signals @@ -24,8 +22,8 @@ @signals.event.sidemenu.connect def _extend_event_menu(sender, **kwargs): - from indico.modules.events.layout.util import MenuEntryData from indico.modules.events.contributions import contribution_settings + from indico.modules.events.layout.util import MenuEntryData def _visible_timetable(event): return contribution_settings.get(event, 'published') diff --git a/indico/modules/events/timetable/blueprint.py b/indico/modules/events/timetable/blueprint.py index 2ddd8ef05c6..9ad3def4c17 100644 --- a/indico/modules/events/timetable/blueprint.py +++ b/indico/modules/events/timetable/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.events.timetable.controllers.display import (RHTimetable, RHTimetableEntryInfo, RHTimetableExportDefaultPDF, RHTimetableExportPDF) from indico.modules.events.timetable.controllers.legacy import (RHLegacyTimetableAddBreak, @@ -32,7 +30,7 @@ _bp = IndicoBlueprint('timetable', __name__, template_folder='templates', virtual_template_folder='events/timetable', - url_prefix='/event/') + url_prefix='/event/') # Management _bp.add_url_rule('/manage/timetable/', 'management', RHManageTimetable) diff --git a/indico/modules/events/timetable/clone.py b/indico/modules/events/timetable/clone.py index 4bd446233e5..395da1e8e57 100644 --- a/indico/modules/events/timetable/clone.py +++ b/indico/modules/events/timetable/clone.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy.orm import defaultload, joinedload from indico.core.db import db diff --git a/indico/modules/events/timetable/controllers/__init__.py b/indico/modules/events/timetable/controllers/__init__.py index 00a5888a296..b37bb592ecf 100644 --- a/indico/modules/events/timetable/controllers/__init__.py +++ b/indico/modules/events/timetable/controllers/__init__.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from enum import Enum from flask import request, session @@ -25,7 +23,7 @@ class SessionManagementLevel(Enum): class RHManageTimetableBase(RHManageEventBase): - """Base class for all timetable management RHs""" + """Base class for all timetable management RHs.""" session_management_level = SessionManagementLevel.none diff --git a/indico/modules/events/timetable/controllers/display.py b/indico/modules/events/timetable/controllers/display.py index 95866edfe50..713da105c1f 100644 --- a/indico/modules/events/timetable/controllers/display.py +++ b/indico/modules/events/timetable/controllers/display.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from io import BytesIO from flask import jsonify, request, session @@ -66,7 +64,6 @@ def _process_args(self): self.entry = self.event.timetable_entries.filter_by(id=request.view_args['entry_id']).first_or_404() def _check_access(self): - RHTimetableProtectionBase._check_access(self) if not self.entry.can_view(session.user): raise Forbidden diff --git a/indico/modules/events/timetable/controllers/display_test.py b/indico/modules/events/timetable/controllers/display_test.py new file mode 100644 index 00000000000..fc9ffb22d3b --- /dev/null +++ b/indico/modules/events/timetable/controllers/display_test.py @@ -0,0 +1,33 @@ +# This file is part of Indico. +# Copyright (C) 2002 - 2021 CERN +# +# Indico is free software; you can redistribute it and/or +# modify it under the terms of the MIT License; see the +# LICENSE file for more details. + +from unittest.mock import MagicMock + +import pytest +from werkzeug.exceptions import Forbidden + +from indico.core.db.sqlalchemy.protection import ProtectionMode +from indico.modules.events.timetable.controllers.display import RHTimetableEntryInfo + + +@pytest.mark.usefixtures('request_context') +@pytest.mark.parametrize('event_allowed', (False, True)) +@pytest.mark.parametrize('allowed', (False, True)) +def test_timetable_entry_info_access(dummy_event, dummy_user, allowed, event_allowed): + dummy_event.protection_mode = ProtectionMode.public if event_allowed else ProtectionMode.protected + rh = RHTimetableEntryInfo() + rh.event = dummy_event + rh.entry = MagicMock() + rh.entry.can_view.return_value = allowed + # event access should not matter for the RH access check as having access e.g. + # to a specific session lets users view the timetable for that session + assert dummy_event.can_access(dummy_user) == event_allowed + if allowed: + rh._check_access() + else: + with pytest.raises(Forbidden): + rh._check_access() diff --git a/indico/modules/events/timetable/controllers/legacy.py b/indico/modules/events/timetable/controllers/legacy.py index ec80a9ad6ef..1229c68cb98 100644 --- a/indico/modules/events/timetable/controllers/legacy.py +++ b/indico/modules/events/timetable/controllers/legacy.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import Counter from datetime import timedelta from operator import attrgetter @@ -308,9 +306,9 @@ def _process(self): data = request.json required_keys = {'contribution_ids', 'day'} allowed_keys = required_keys | {'session_block_id'} - if set(data.viewkeys()) > allowed_keys: + if set(data.keys()) > allowed_keys: raise BadRequest('Invalid keys found') - elif required_keys > set(data.viewkeys()): + elif required_keys > set(data.keys()): raise BadRequest('Required keys missing') entries = [] day = dateutil.parser.parse(data['day']).date() @@ -405,7 +403,7 @@ def _process(self): class RHLegacyTimetableMoveEntry(RHManageTimetableEntryBase): - """Moves a TimetableEntry into a Session or top-level timetable""" + """Move a TimetableEntry into a Session or top-level timetable.""" def _process_GET(self): current_day = dateutil.parser.parse(request.args.get('day')).date() @@ -483,7 +481,7 @@ def _process(self): class RHLegacyTimetableEditEntryDateTime(RHManageTimetableEntryBase): - """Changes the start_dt of a `TimetableEntry`""" + """Change the start_dt of a `TimetableEntry`.""" @property def session_management_level(self): diff --git a/indico/modules/events/timetable/controllers/manage.py b/indico/modules/events/timetable/controllers/manage.py index 9f18b645fcc..306e2dbaabc 100644 --- a/indico/modules/events/timetable/controllers/manage.py +++ b/indico/modules/events/timetable/controllers/manage.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import dateutil.parser from flask import jsonify, request, session from werkzeug.exceptions import BadRequest, Forbidden, NotFound @@ -31,7 +29,7 @@ class RHManageTimetable(RHManageTimetableBase): - """Display timetable management page""" + """Display timetable management page.""" session_management_level = SessionManagementLevel.coordinate @@ -63,7 +61,7 @@ def _process(self): class RHTimetableREST(RHManageTimetableEntryBase): - """RESTful timetable actions""" + """RESTful timetable actions.""" def _get_contribution_updates(self, data): updates = {'parent': None} @@ -89,13 +87,13 @@ def _get_contribution_updates(self, data): return updates def _process_POST(self): - """Create new timetable entry""" + """Create new timetable entry.""" data = request.json required_keys = {'start_dt'} allowed_keys = {'start_dt', 'contribution_id', 'session_block_id', 'force'} - if set(data.viewkeys()) > allowed_keys: + if set(data.keys()) > allowed_keys: raise BadRequest('Invalid keys found') - elif required_keys > set(data.viewkeys()): + elif required_keys > set(data.keys()): raise BadRequest('Required keys missing') updates = {'start_dt': dateutil.parser.parse(data['start_dt'])} if 'contribution_id' in data: @@ -107,10 +105,10 @@ def _process_POST(self): return jsonify(start_dt=entry.start_dt.isoformat(), id=entry.id) def _process_PATCH(self): - """Update a timetable entry""" + """Update a timetable entry.""" data = request.json # TODO: support breaks - if set(data.viewkeys()) > {'start_dt'}: + if set(data.keys()) > {'start_dt'}: raise BadRequest('Invalid keys found') updates = {} if 'start_dt' in data: @@ -121,7 +119,7 @@ def _process_PATCH(self): return jsonify() def _process_DELETE(self): - """Delete a timetable entry""" + """Delete a timetable entry.""" if self.entry.type == TimetableEntryType.SESSION_BLOCK: delete_session_block(self.entry.session_block) elif self.event.type != 'conference' and self.entry.type == TimetableEntryType.CONTRIBUTION: @@ -163,7 +161,7 @@ def _check_access(self): def _process_PATCH(self): data = request.json - if set(data.viewkeys()) > {'colors'}: + if set(data.keys()) > {'colors'}: raise BadRequest if 'colors' in data: colors = ColorTuple(**data['colors']) @@ -173,7 +171,7 @@ def _process_PATCH(self): class RHCloneContribution(RHManageTimetableBase): - """Clone a contribution and schedule it at the same position""" + """Clone a contribution and schedule it at the same position.""" def _process_args(self): RHManageTimetableBase._process_args(self) diff --git a/indico/modules/events/timetable/forms.py b/indico/modules/events/timetable/forms.py index f770451dc7f..37f11d242a9 100644 --- a/indico/modules/events/timetable/forms.py +++ b/indico/modules/events/timetable/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import datetime, timedelta from flask import request @@ -30,7 +28,7 @@ from indico.web.forms.widgets import SwitchWidget -class EntryFormMixin(object): +class EntryFormMixin: _entry_type = None _default_duration = None _display_fields = None @@ -52,11 +50,11 @@ def __init__(self, *args, **kwargs): else: defaults.duration = self._default_duration kwargs['obj'] = defaults - super(EntryFormMixin, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) @property def data(self): - data = super(EntryFormMixin, self).data + data = super().data del data['time'] return data @@ -106,7 +104,7 @@ class ContributionEntryForm(EntryFormMixin, ContributionForm): def __init__(self, *args, **kwargs): kwargs['to_schedule'] = kwargs.get('to_schedule', True) - super(ContributionEntryForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) @property def _default_duration(self): @@ -126,7 +124,7 @@ def _validate_duration(entry, field, start_dt): raise ValidationError(_("This duration is too short to fit the entries within.")) def validate_duration(self, field): - super(SessionBlockEntryForm, self).validate_duration(field) + super().validate_duration(field) if self.session_block and self.start_dt.data: self._validate_duration(self.session_block.timetable_entry, field, self.start_dt) @@ -138,10 +136,10 @@ class BaseEntryForm(EntryFormMixin, IndicoForm): def __init__(self, *args, **kwargs): self.entry = kwargs.pop('entry') self._entry_type = self.entry.type - super(BaseEntryForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) def validate_duration(self, field): - super(BaseEntryForm, self).validate_duration(field) + super().validate_duration(field) if self.entry.type == TimetableEntryType.SESSION_BLOCK and self.entry.children: SessionBlockEntryForm._validate_duration(self.entry, field, self.start_dt) diff --git a/indico/modules/events/timetable/legacy.py b/indico/modules/events/timetable/legacy.py index a092e6ade96..5d545d4abd7 100644 --- a/indico/modules/events/timetable/legacy.py +++ b/indico/modules/events/timetable/legacy.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import defaultdict from hashlib import md5 from itertools import chain @@ -21,7 +19,7 @@ from indico.web.flask.util import url_for -class TimetableSerializer(object): +class TimetableSerializer: def __init__(self, event, management=False, user=None): self.management = management self.user = user if user is not None or not has_request_context() else session.user @@ -54,7 +52,7 @@ def serialize_timetable(self, days=None, hide_weekends=False, strip_empty_days=F data = self.serialize_timetable_entry(entry, load_children=False) key = self._get_entry_key(entry) if entry.parent: - parent_code = 's{}'.format(entry.parent_id) + parent_code = f's{entry.parent_id}' timetable[date_str][parent_code]['entries'][key] = data else: if (entry.type == TimetableEntryType.SESSION_BLOCK and @@ -160,11 +158,11 @@ def serialize_contribution_entry(self, entry): 'description': contribution.description, 'duration': contribution.duration.seconds / 60, 'pdf': url_for('contributions.export_pdf', entry.contribution), - 'presenters': map(self._get_person_data, - sorted([p for p in contribution.person_links if p.is_speaker], - key=lambda x: (x.author_type != AuthorType.primary, - x.author_type != AuthorType.secondary, - x.display_order_key))), + 'presenters': list(map(self._get_person_data, + sorted([p for p in contribution.person_links if p.is_speaker], + key=lambda x: (x.author_type != AuthorType.primary, + x.author_type != AuthorType.secondary, + x.display_order_key)))), 'sessionCode': block.session.code if block else None, 'sessionId': block.session_id if block else None, 'sessionSlotId': block.id if block else None, @@ -172,7 +170,7 @@ def serialize_contribution_entry(self, entry): 'title': contribution.title, 'url': url_for('contributions.display_contribution', contribution), 'friendlyId': contribution.friendly_id, - 'references': map(SerializerBase.serialize_reference, contribution.references), + 'references': list(map(SerializerBase.serialize_reference, contribution.references)), 'board_number': contribution.board_number}) return data @@ -208,12 +206,12 @@ def serialize_folder(folder): '_type': 'AttachmentFolder', '_fossil': 'folder', 'title': folder.title, - 'attachments': map(serialize_attachment, folder.attachments)} + 'attachments': list(map(serialize_attachment, folder.attachments))} data = {'files': [], 'folders': []} items = obj.attached_items - data['files'] = map(serialize_attachment, items.get('files', [])) - data['folders'] = map(serialize_folder, items.get('folders', [])) + data['files'] = list(map(serialize_attachment, items.get('files', []))) + data['folders'] = list(map(serialize_folder, items.get('folders', []))) if not data['files'] and not data['folders']: data['files'] = None return data @@ -247,11 +245,11 @@ def _get_entry_data(self, entry): def _get_entry_key(self, entry): if entry.type == TimetableEntryType.SESSION_BLOCK: - return 's{}'.format(entry.id) + return f's{entry.id}' elif entry.type == TimetableEntryType.CONTRIBUTION: - return 'c{}'.format(entry.id) + return f'c{entry.id}' elif entry.type == TimetableEntryType.BREAK: - return 'b{}'.format(entry.id) + return f'b{entry.id}' else: raise ValueError() @@ -275,7 +273,7 @@ def _get_person_data(self, person_link): data = {'firstName': person_link.first_name, 'familyName': person_link.last_name, 'affiliation': person_link.affiliation, - 'emailHash': md5(person.email.encode('utf-8')).hexdigest() if person.email else None, + 'emailHash': md5(person.email.encode()).hexdigest() if person.email else None, 'name': person_link.get_full_name(last_name_first=False, last_name_upper=False, abbrev_first_name=False, show_title=True), 'displayOrderKey': person_link.display_order_key} @@ -312,7 +310,7 @@ def serialize_entry_update(entry, session_=None): def serialize_event_info(event): return {'_type': 'Conference', - 'id': unicode(event.id), + 'id': str(event.id), 'title': event.title, 'startDate': event.start_dt_local, 'endDate': event.end_dt_local, @@ -321,7 +319,7 @@ def serialize_event_info(event): def serialize_session(sess): - """Return data for a single session""" + """Return data for a single session.""" data = { '_type': 'Session', 'address': sess.address, diff --git a/indico/modules/events/timetable/models/breaks.py b/indico/modules/events/timetable/models/breaks.py index d078f0c02de..9ee3b064953 100644 --- a/indico/modules/events/timetable/models/breaks.py +++ b/indico/modules/events/timetable/models/breaks.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from sqlalchemy import DDL from sqlalchemy.event import listens_for from sqlalchemy.ext.declarative import declared_attr @@ -18,7 +16,7 @@ from indico.core.db.sqlalchemy.locations import LocationMixin from indico.core.db.sqlalchemy.util.models import auto_table_args from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class Break(DescriptionMixin, ColorMixin, LocationMixin, db.Model): @@ -74,7 +72,6 @@ def start_dt(self): def end_dt(self): return self.timetable_entry.start_dt + self.duration if self.timetable_entry else None - @return_ascii def __repr__(self): return format_repr(self, 'id', _text=self.title) diff --git a/indico/modules/events/timetable/models/entries.py b/indico/modules/events/timetable/models/entries.py index 53cb85f64a5..29739c0e562 100644 --- a/indico/modules/events/timetable/models/entries.py +++ b/indico/modules/events/timetable/models/entries.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import timedelta from sqlalchemy import DDL @@ -19,10 +17,10 @@ from indico.core.db.sqlalchemy import PyIntEnum, UTCDateTime from indico.core.db.sqlalchemy.util.models import populate_one_to_one_backrefs from indico.util.date_time import overlaps +from indico.util.enum import RichIntEnum from indico.util.i18n import _ from indico.util.locators import locator_property -from indico.util.string import format_repr, return_ascii -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_repr class TimetableEntryType(RichIntEnum): @@ -37,10 +35,10 @@ def _make_check(type_, *cols): all_cols = {'session_block_id', 'contribution_id', 'break_id'} required_cols = all_cols & set(cols) forbidden_cols = all_cols - required_cols - criteria = ['{} IS NULL'.format(col) for col in sorted(forbidden_cols)] - criteria += ['{} IS NOT NULL'.format(col) for col in sorted(required_cols)] + criteria = [f'{col} IS NULL' for col in sorted(forbidden_cols)] + criteria += [f'{col} IS NOT NULL' for col in sorted(required_cols)] condition = 'type != {} OR ({})'.format(type_, ' AND '.join(criteria)) - return db.CheckConstraint(condition, 'valid_{}'.format(type_.name.lower())) + return db.CheckConstraint(condition, f'valid_{type_.name.lower()}') class TimetableEntry(db.Model): @@ -52,7 +50,7 @@ def __table_args__(cls): _make_check(TimetableEntryType.SESSION_BLOCK, 'session_block_id'), _make_check(TimetableEntryType.CONTRIBUTION, 'contribution_id'), _make_check(TimetableEntryType.BREAK, 'break_id'), - db.CheckConstraint("type != {} OR parent_id IS NULL".format(TimetableEntryType.SESSION_BLOCK), + db.CheckConstraint(f"type != {TimetableEntryType.SESSION_BLOCK} OR parent_id IS NULL", 'valid_parent'), {'schema': 'events'}) @@ -180,7 +178,7 @@ def object(self, value): elif isinstance(value, Break): self.break_ = value elif value is not None: - raise TypeError('Unexpected object: {}'.format(value)) + raise TypeError(f'Unexpected object: {value}') @hybrid_property def duration(self): @@ -200,17 +198,17 @@ def duration(cls): db.select([SessionBlock.duration]) .where(SessionBlock.id == cls.session_block_id) .correlate_except(SessionBlock) - .as_scalar(), + .scalar_subquery(), TimetableEntryType.CONTRIBUTION.value: db.select([Contribution.duration]) .where(Contribution.id == cls.contribution_id) .correlate_except(Contribution) - .as_scalar(), + .scalar_subquery(), TimetableEntryType.BREAK.value: db.select([Break.duration]) .where(Break.id == cls.break_id) .correlate_except(Break) - .as_scalar(), + .scalar_subquery(), }, value=cls.type) @hybrid_property @@ -235,7 +233,7 @@ def session_siblings(self): @property def siblings(self): - from indico.modules.events.timetable.util import get_top_level_entries, get_nested_entries + from indico.modules.events.timetable.util import get_nested_entries, get_top_level_entries tzinfo = self.event.tzinfo day = self.start_dt.astimezone(tzinfo).date() siblings = (get_nested_entries(self.event)[self.parent_id] @@ -256,12 +254,11 @@ def siblings_query(self): def locator(self): return dict(self.event.locator, entry_id=self.id) - @return_ascii def __repr__(self): return format_repr(self, 'id', 'type', 'start_dt', 'end_dt', _repr=self.object) def can_view(self, user): - """Checks whether the user will see this entry in the timetable.""" + """Check whether the user will see this entry in the timetable.""" if self.type in (TimetableEntryType.CONTRIBUTION, TimetableEntryType.BREAK): return self.object.can_access(user) elif self.type == TimetableEntryType.SESSION_BLOCK: diff --git a/indico/modules/events/timetable/operations.py b/indico/modules/events/timetable/operations.py index 633acd10e75..1332f4d2578 100644 --- a/indico/modules/events/timetable/operations.py +++ b/indico/modules/events/timetable/operations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from operator import attrgetter from flask import session @@ -75,7 +73,7 @@ def create_timetable_entry(event, data, parent=None, extend_parent=False): signals.event.timetable_entry_created.send(entry) logger.info('Timetable entry %s created by %s', entry, user) entry.event.log(EventLogRealm.management, EventLogKind.positive, 'Timetable', - "Entry for {} '{}' created".format(object_type, object_title), user, + f"Entry for {object_type} '{object_title}' created", user, data={'Time': format_datetime(entry.start_dt, timezone=event.tzinfo)}) if extend_parent: entry.extend_parent() @@ -101,7 +99,7 @@ def update_timetable_entry(entry, data): signals.event.timetable_entry_updated.send(entry, changes=changes) logger.info('Timetable entry %s updated by %s', entry, session.user) entry.event.log(EventLogRealm.management, EventLogKind.change, 'Timetable', - "Entry for {} '{}' modified".format(object_type, object_title), session.user, + f"Entry for {object_type} '{object_title}' modified", session.user, data={'Time': format_datetime(entry.start_dt)}) @@ -113,7 +111,7 @@ def delete_timetable_entry(entry, log=True): if log: logger.info('Timetable entry %s deleted by %s', entry, session.user) entry.event.log(EventLogRealm.management, EventLogKind.negative, 'Timetable', - "Entry for {} '{}' deleted".format(object_type, object_title), session.user, + f"Entry for {object_type} '{object_title}' deleted", session.user, data={'Time': format_datetime(entry.start_dt)}) @@ -133,7 +131,7 @@ def fit_session_block_entry(entry, log=True): def move_timetable_entry(entry, parent=None, day=None): - """Move the `entry` to another session or top-level timetable + """Move the `entry` to another session or top-level timetable. :param entry: `TimetableEntry` to be moved :param parent: If specified then the entry will be set as a child @@ -171,7 +169,7 @@ def move_timetable_entry(entry, parent=None, day=None): def update_timetable_entry_object(entry, data): - """Update the `object` of a timetable entry according to its type""" + """Update the `object` of a timetable entry according to its type.""" from indico.modules.events.contributions.operations import update_contribution obj = entry.object if entry.type == TimetableEntryType.CONTRIBUTION: @@ -184,7 +182,7 @@ def update_timetable_entry_object(entry, data): def swap_timetable_entry(entry, direction, session_=None): - """Swap entry with closest gap or non-parallel sibling""" + """Swap entry with closest gap or non-parallel sibling.""" in_session = session_ is not None sibling = get_sibling_entry(entry, direction=direction, in_session=in_session) if not sibling: diff --git a/indico/modules/events/timetable/reschedule.py b/indico/modules/events/timetable/reschedule.py index ef21e35fc03..83f1b3c77ae 100644 --- a/indico/modules/events/timetable/reschedule.py +++ b/indico/modules/events/timetable/reschedule.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from datetime import datetime, timedelta from flask import session @@ -21,12 +19,12 @@ from indico.modules.events.timetable.models.entries import TimetableEntry, TimetableEntryType from indico.modules.events.timetable.operations import fit_session_block_entry from indico.util.date_time import format_date, format_human_timedelta +from indico.util.enum import RichEnum from indico.util.i18n import _ -from indico.util.struct.enum import RichEnum -from indico.util.struct.iterables import materialize_iterable, window +from indico.util.iterables import materialize_iterable, window -class RescheduleMode(unicode, RichEnum): +class RescheduleMode(str, RichEnum): __titles__ = {'none': 'Fit blocks', 'time': 'Start times', 'duration': 'Durations'} none = 'none' # no action, just fit blocks.. time = 'time' @@ -37,9 +35,9 @@ def title(self): return RichEnum.title.fget(self) -class Rescheduler(object): +class Rescheduler: """ - Compacts the the schedule of an event day by either adjusting + Compact the the schedule of an event day by either adjusting start times or durations of timetable entries. :param event: The event of which the timetable entries should @@ -74,7 +72,7 @@ def __init__(self, event, mode, day, session=None, session_block=None, fit_block self.gap = gap def run(self): - """Perform the rescheduling""" + """Perform the rescheduling.""" if self.fit_blocks: self._fit_blocks() if self.mode == RescheduleMode.time: diff --git a/indico/modules/events/timetable/templates/balloons/contribution.html b/indico/modules/events/timetable/templates/balloons/contribution.html index db18271c728..a30341b0767 100644 --- a/indico/modules/events/timetable/templates/balloons/contribution.html +++ b/indico/modules/events/timetable/templates/balloons/contribution.html @@ -69,7 +69,7 @@ {% endif %} {% endif %} - {% if (editable and can_manage_contributions) or subcontribution_count %} + {% if (editable and can_manage_contributions) or contrib.subcontribution_count %}
      diff --git a/indico/modules/events/timetable/templates/display/_weeks.html b/indico/modules/events/timetable/templates/display/_weeks.html index 6e287fa8cbd..faa54298705 100644 --- a/indico/modules/events/timetable/templates/display/_weeks.html +++ b/indico/modules/events/timetable/templates/display/_weeks.html @@ -50,7 +50,7 @@ {% if entry.type.name == 'CONTRIBUTION' and speakers %}
        {% for link in speakers -%} -
      • {{ render_user_data(link, show_title=false, italic_affiliation=italic_affiliation) }}
        • +
      • {{ render_user_data(link, show_title=false) }}
        • {% endfor %}
      {% endif %} @@ -75,7 +75,7 @@
      {{ start_time|format_time(timezone=timezone) if loop.first else '' }} - {% if hide_placeholders and show_end_times %} + {% if show_end_times %} {{ entry.end_dt | format_time(timezone=timezone) }} @@ -118,7 +118,7 @@ {% endif %} {% endif %} {% if height > 0 %} -
      +
      {% endif %} {% endmacro %} @@ -136,7 +136,10 @@ {% endif %} {% set has_multi = false %} + {% set same_time = true %} + {% set extra_styles = [] %} {% if loop.first %} + {% set same_time = false %} {% set time_parts = (entry.duration.seconds/60) /5 %} {% set height = (time_parts * px_per_5_minutes) %} {% if to_subtract[-1] %} @@ -174,8 +177,6 @@ {% set __ = extra_styles.append('background-color: #%s'|format(session.background_color)) %} {% endif %} {% endif %} - {% else %} - {% set same_time = true %} {% endif %} {% if entry.type.name == 'CONTRIBUTION' or session %} {% if entry.type.name == 'CONTRIBUTION' %} @@ -210,9 +211,9 @@ {{ day | format_date(format='full', timezone=timezone) }}
      {% set hidden_count = [] %} - {% for start_time, slot_entries in day_entries.viewitems() %} + {% for start_time, slot_entries in day_entries.items() %} {% set start_time_minutes = (start_time.strftime('%-H')|int*60 + start_time.strftime('%-M')|int) %} - {% set next_entry = day_entries.keys()[loop.index] %} + {% set next_entry = (day_entries.keys()|list)[loop.index] %} {% if next_entry %} {% set next_start_minutes = (next_entry.strftime('%-H')|int*60 + next_entry.strftime('%-M')|int)%} {% else %} diff --git a/indico/modules/events/timetable/templates/display/indico/_contribution.html b/indico/modules/events/timetable/templates/display/indico/_contribution.html index 0bc179d9841..6eb07920950 100644 --- a/indico/modules/events/timetable/templates/display/indico/_contribution.html +++ b/indico/modules/events/timetable/templates/display/indico/_contribution.html @@ -6,8 +6,7 @@ {% macro render_contribution(contrib, event, theme_settings, theme_context, parent=none, nested=false, hide_end_time=false, timezone=none, show_notes=false, show_location=false) -%} - {% set anchor = slugify(contrib.friendly_id, contrib.title, maxlen=30) %} -
    • +
    • {% if theme_settings.number_contributions %} @@ -20,7 +19,7 @@
      - + {{- contrib.title -}} {% if contrib.duration and not theme_settings.hide_duration -%} diff --git a/indico/modules/events/timetable/templates/display/indico/_subcontribution.html b/indico/modules/events/timetable/templates/display/indico/_subcontribution.html index ed8995c024b..5f17cf80383 100644 --- a/indico/modules/events/timetable/templates/display/indico/_subcontribution.html +++ b/indico/modules/events/timetable/templates/display/indico/_subcontribution.html @@ -4,15 +4,13 @@ {% macro render_subcontribution(subcontrib, event, theme_settings, theme_context, show_notes=true) %} - {% set contrib = subcontrib.contribution %} - {% set anchor = slugify('sc', contrib.friendly_id, subcontrib.friendly_id, subcontrib.title, maxlen=30) %} -
    • +
    • - + {% if theme_settings.number_contributions %} - {{- theme_context.num_contribution }}. - {{- theme_context.num_subcontribution }} + {%- set n_scontrib = theme_context.num_subcontribution -%} + {{- 'abcdefghijklmnopqrstuvwxyz'[n_scontrib % 28 - 1] * (n_scontrib / 28 + 1)|int }}) {% endif %} {{- subcontrib.title -}} diff --git a/indico/modules/events/timetable/templates/move_entry.html b/indico/modules/events/timetable/templates/move_entry.html index 1258cb5e725..7187baaa835 100644 --- a/indico/modules/events/timetable/templates/move_entry.html +++ b/indico/modules/events/timetable/templates/move_entry.html @@ -6,7 +6,7 @@ {% endif %}
      - {% for day, entries in top_level_entries.iteritems() | sort %} + {% for day, entries in top_level_entries.items() | sort %}
      @@ -64,13 +64,15 @@
      {# Do not translate
      since they are the official terms in OAuth2 RFC #}
      Client ID
      {#--#} -
      {{ application.client_id }}
      +
      {{ clipboard_text(application.client_id) }}
      Client Secret
      {#--#} -
      {{ application.client_secret }}
      +
      {{ clipboard_text(application.client_secret) }}
      Authorize URL
      {#--#} -
      {{ url_for('.oauth_authorize', _external=true) }}
      +
      {{ clipboard_text(url_for('.oauth_authorize', _external=true)) }}
      Access token URL
      {#--#} -
      {{ url_for('.oauth_token', _external=true) }}
      +
      {{ clipboard_text(url_for('.oauth_token', _external=true)) }}
      +
      OAuth metadata
      {#--#} +
      {{ clipboard_text(url_for('.oauth_metadata', _external=true), as_link=true) }}
      {{ form_header(form) }} @@ -83,3 +85,16 @@
      {% endblock %} + +{% macro clipboard_text(url, as_link=false) %} + {% if as_link %} +       {{ url }} + {% else %} + {{ url }} + {% endif %} + +{% endmacro %} diff --git a/indico/modules/oauth/templates/user_profile.html b/indico/modules/oauth/templates/user_profile.html index 3d984c804ac..017210a66cd 100644 --- a/indico/modules/oauth/templates/user_profile.html +++ b/indico/modules/oauth/templates/user_profile.html @@ -11,29 +11,32 @@

      {% trans %}This is the list of applications you authorized to access your Indico data.{% endtrans %}

        - {% if not tokens %} + {% if not authorizations %}
      • {%- trans %}No third party applications have been authorized.{% endtrans -%}
        • {% endif %} - {% for token in tokens|sort(attribute='application.name') %} + {% for auth, last_use in authorizations %}
      • - {{ token.application.name }} + + {{- auth.application.name -}} + {% trans %}Last used:{% endtrans %} - {% if token.last_used_dt %} - {{ token.last_used_dt | format_datetime('short') }} + {% if last_use %} + {{ last_use | format_datetime('short') }} {% else %} {% trans %}Never{% endtrans %} {% endif %} diff --git a/indico/modules/oauth/testing/__init__.py b/indico/modules/oauth/testing/__init__.py deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/indico/modules/oauth/testing/fixtures.py b/indico/modules/oauth/testing/fixtures.py deleted file mode 100644 index a457c250185..00000000000 --- a/indico/modules/oauth/testing/fixtures.py +++ /dev/null @@ -1,59 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from uuid import uuid4 - -import pytest - -from indico.modules.oauth.models.applications import OAuthApplication -from indico.modules.oauth.models.tokens import OAuthToken - - -@pytest.fixture -def create_application(db): - """Returns a callable which lets you create applications""" - - def _create_application(name, **params): - params.setdefault('client_id', unicode(uuid4())) - params.setdefault('default_scopes', 'read:user') - params.setdefault('redirect_uris', 'http://localhost:10500') - params.setdefault('is_trusted', True) - application = OAuthApplication(name=name, **params) - db.session.add(application) - db.session.flush() - return application - - return _create_application - - -@pytest.fixture -def create_token(db, dummy_application, dummy_user): - """Returns a callable which lets you create tokens""" - - def _create_tokens(**params): - params.setdefault('access_token', unicode(uuid4())) - params.setdefault('user', dummy_user) - params.setdefault('application', dummy_application) - params.setdefault('scopes', ['read:api', 'write:api']) - token = OAuthToken(**params) - db.session.add(token) - db.session.flush() - return token - - return _create_tokens - - -@pytest.fixture -def dummy_application(create_application): - """Gives you a dummy application""" - return create_application(name='dummy') - - -@pytest.fixture -def dummy_token(create_token): - """Returns a dummy token""" - return create_token() diff --git a/indico/modules/oauth/views.py b/indico/modules/oauth/views.py index 1ff481f068d..d0c30e7a6e4 100644 --- a/indico/modules/oauth/views.py +++ b/indico/modules/oauth/views.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.modules.admin.views import WPAdmin from indico.modules.users.views import WPUser from indico.web.views import WPJinjaMixin diff --git a/indico/modules/rb/__init__.py b/indico/modules/rb/__init__.py index bda117a0b0c..e366d3f9e5f 100644 --- a/indico/modules/rb/__init__.py +++ b/indico/modules/rb/__init__.py @@ -1,15 +1,14 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from flask import session from indico.core import signals +from indico.core.cache import make_scoped_cache from indico.core.config import config from indico.core.logger import Logger from indico.core.permissions import ManagementPermission, check_permissions @@ -23,6 +22,7 @@ logger = Logger.get('rb') +rb_cache = make_scoped_cache('roombooking') rb_settings = SettingsProxy('roombooking', { diff --git a/indico/modules/rb/api.py b/indico/modules/rb/api.py index 314bd65f3e7..ed6316c82d2 100644 --- a/indico/modules/rb/api.py +++ b/indico/modules/rb/api.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -34,7 +34,7 @@ class RoomBookingHookBase(HTTPAPIHook): GUEST_ALLOWED = False def _getParams(self): - super(RoomBookingHookBase, self)._getParams() + super()._getParams() self._fromDT = utc_to_server(self._fromDT.astimezone(pytz.utc)).replace(tzinfo=None) if self._fromDT else None self._toDT = utc_to_server(self._toDT.astimezone(pytz.utc)).replace(tzinfo=None) if self._toDT else None self._occurrences = _yesno(get_query_parameter(self._queryParams, ['occ', 'occurrences'], 'no')) @@ -56,9 +56,9 @@ class RoomHook(RoomBookingHookBase): VALID_FORMATS = ('json', 'jsonp', 'xml') def _getParams(self): - super(RoomHook, self)._getParams() + super()._getParams() self._location = self._pathParams['location'] - self._ids = map(int, self._pathParams['idlist'].split('-')) + self._ids = list(map(int, self._pathParams['idlist'].split('-'))) if self._detail not in {'rooms', 'reservations'}: raise HTTPAPIError('Invalid detail level: %s' % self._detail, 400) @@ -94,7 +94,7 @@ class RoomNameHook(RoomBookingHookBase): VALID_FORMATS = ('json', 'jsonp', 'xml') def _getParams(self): - super(RoomNameHook, self)._getParams() + super()._getParams() self._location = self._pathParams['location'] self._room_name = self._pathParams['room_name'] @@ -107,7 +107,7 @@ def export_roomName(self, user): if loc is None: return - search_str = '%{}%'.format(self._room_name) + search_str = f'%{self._room_name}%' rooms_data = Room.get_with_data( filters=[ Room.location_id == loc.id, @@ -135,7 +135,7 @@ def serializer_args(self): return {'ical_serializer': _ical_serialize_reservation} def _getParams(self): - super(ReservationHook, self)._getParams() + super()._getParams() self._locations = self._pathParams['loclist'].split('-') def export_reservation(self, user): @@ -158,7 +158,7 @@ class BookRoomHook(HTTPAPIHook): HTTP_POST = True def _getParams(self): - super(BookRoomHook, self)._getParams() + super()._getParams() self._fromDT = utc_to_server(self._fromDT.astimezone(pytz.utc)).replace(tzinfo=None) if self._fromDT else None self._toDT = utc_to_server(self._toDT.astimezone(pytz.utc)).replace(tzinfo=None) if self._toDT else None if not self._fromDT or not self._toDT or self._fromDT.date() != self._toDT.date(): @@ -175,7 +175,7 @@ def _getParams(self): if not users: raise HTTPAPIError('Username does not exist') elif len(users) != 1: - raise HTTPAPIError('Ambiguous username ({} users found)'.format(len(users))) + raise HTTPAPIError(f'Ambiguous username ({len(users)} users found)') user = users[0] self._params = { @@ -185,7 +185,7 @@ def _getParams(self): 'from': self._fromDT, 'to': self._toDT } - missing = [key for key, val in self._params.iteritems() if not val] + missing = [key for key, val in self._params.items() if not val] if missing: raise HTTPAPIError('Required params missing: {}'.format(', '.join(missing))) self._room = Room.get(self._params['room_id']) @@ -216,14 +216,14 @@ def api_roomBooking(self, user): except ConflictingOccurrences: raise HTTPAPIError('Failed to create the booking due to conflicts with other bookings') except IndicoError as e: - raise HTTPAPIError('Failed to create the booking: {}'.format(e)) + raise HTTPAPIError(f'Failed to create the booking: {e}') db.session.add(reservation) db.session.flush() return {'reservationID': reservation.id} def _export_reservations(hook, limit_per_room, include_rooms, extra_filters=None): - """Exports reservations. + """Export reservations. :param hook: The HTTPAPIHook instance :param limit_per_room: Should the limit/offset be applied per room @@ -243,7 +243,7 @@ def _export_reservations(hook, limit_per_room, include_rooms, extra_filters=None filters.append(cast(Reservation.start_dt, Time) >= hook._fromDT.time()) filters += _get_reservation_state_filter(hook._queryParams) occurs = [datetime.strptime(x, '%Y-%m-%d').date() - for x in filter(None, get_query_parameter(hook._queryParams, ['occurs'], '').split(','))] + for x in [_f for _f in get_query_parameter(hook._queryParams, ['occurs'], '').split(',') if _f]] data = [] if hook._occurrences: data.append('occurrences') @@ -260,7 +260,7 @@ def _export_reservations(hook, limit_per_room, include_rooms, extra_filters=None def _serializable_room(room_data, reservations=None): - """Serializable room data + """Serializable room data. :param room_data: Room data :param reservations: MultiDict mapping for room id => reservations @@ -273,7 +273,7 @@ def _serializable_room(room_data, reservations=None): def _serializable_room_minimal(room): - """Serializable minimal room data (inside reservations) + """Serializable minimal room data (inside reservations). :param room: A `Room` """ @@ -283,7 +283,7 @@ def _serializable_room_minimal(room): def _serializable_reservation(reservation_data, include_room=False): - """Serializable reservation (standalone or inside room) + """Serializable reservation (standalone or inside room). :param reservation_data: Reservation data :param include_room: Include minimal room information @@ -315,7 +315,7 @@ def _ical_serialize_repeatability(data): recur['interval'] = data['repeat_interval'] elif data['repeat_frequency'] == RepeatFrequency.MONTH: recur['freq'] = 'monthly' - recur['byday'] = '{}{}'.format(start_dt_utc.day // 7, WEEK_DAYS[start_dt_utc.weekday()]) + recur['byday'] = f'{start_dt_utc.day // 7}{WEEK_DAYS[start_dt_utc.weekday()]}' return recur @@ -330,8 +330,8 @@ def _ical_serialize_reservation(cal, data, now): event.add('dtend', end_dt_utc) event.add('url', data['bookingUrl']) event.add('summary', data['reason']) - event.add('location', u'{}: {}'.format(data['location'], data['room']['fullName'])) - event.add('description', data['reason'].decode('utf-8') + '\n\n' + data['bookingUrl']) + event.add('location', '{}: {}'.format(data['location'], data['room']['fullName'])) + event.add('description', data['reason'] + '\n\n' + data['bookingUrl']) if data['repeat_frequency'] != RepeatFrequency.NEVER: event.add('rrule', _ical_serialize_repeatability(data)) cal.add_component(event) diff --git a/indico/modules/rb/blueprint.py b/indico/modules/rb/blueprint.py index 8457be63536..d7021da9c7a 100644 --- a/indico/modules/rb/blueprint.py +++ b/indico/modules/rb/blueprint.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import json from flask import jsonify, redirect @@ -121,22 +119,22 @@ _bp.add_url_rule('/api/admin/map-areas', 'admin_map_areas', admin.RHMapAreas, methods=('POST', 'PATCH', 'DELETE')) # Event linking -_bp.add_url_rule('!/event//manage/rooms/', 'event_booking_list', event.RHEventBookingList) -_bp.add_url_rule('!/event//manage/rooms/linking/contributions', 'event_linkable_contributions', +_bp.add_url_rule('!/event//manage/rooms/', 'event_booking_list', event.RHEventBookingList) +_bp.add_url_rule('!/event//manage/rooms/linking/contributions', 'event_linkable_contributions', event.RHListLinkableContributions) -_bp.add_url_rule('!/event//manage/rooms/linking/session-blocks', 'event_linkable_session_blocks', +_bp.add_url_rule('!/event//manage/rooms/linking/session-blocks', 'event_linkable_session_blocks', event.RHListLinkableSessionBlocks) # Deep/quick links @_bp.route('/room/') def room_link(room_id): - return redirect(url_for('rb.roombooking', modal='room-details:{}'.format(room_id))) + return redirect(url_for('rb.roombooking', modal=f'room-details:{room_id}')) @_bp.route('/booking/') def booking_link(booking_id): - return redirect(url_for('rb.roombooking', modal='booking-details:{}'.format(booking_id))) + return redirect(url_for('rb.roombooking', modal=f'booking-details:{booking_id}')) @_bp.route('/booking//cancel/') @@ -152,7 +150,7 @@ def my_bookings_link(): @_bp.route('/blocking/') def blocking_link(blocking_id): - return redirect(url_for('rb.roombooking', path='blockings', modal='blocking-details:{}'.format(blocking_id))) + return redirect(url_for('rb.roombooking', path='blockings', modal=f'blocking-details:{blocking_id}')) _compat_bp = IndicoBlueprint('compat_rb', __name__, url_prefix='/rooms') diff --git a/indico/modules/rb/client/js/actions.js b/indico/modules/rb/client/js/actions.js index 04e726357a6..6258d17b818 100644 --- a/indico/modules/rb/client/js/actions.js +++ b/indico/modules/rb/client/js/actions.js @@ -1,12 +1,13 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import qs from 'qs'; import {push} from 'connected-react-router'; +import qs from 'qs'; + import {history} from './history'; // Page state diff --git a/indico/modules/rb/client/js/common/bookings/BookingDetails.jsx b/indico/modules/rb/client/js/common/bookings/BookingDetails.jsx index 5011d7f1c35..2b034eaccc0 100644 --- a/indico/modules/rb/client/js/common/bookings/BookingDetails.jsx +++ b/indico/modules/rb/client/js/common/bookings/BookingDetails.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -9,11 +9,11 @@ import bookingLinkURL from 'indico-url:rb.booking_link'; import _ from 'lodash'; import moment from 'moment'; -import React from 'react'; import PropTypes from 'prop-types'; +import React from 'react'; import {Form as FinalForm} from 'react-final-form'; -import {bindActionCreators} from 'redux'; import {connect} from 'react-redux'; +import {bindActionCreators} from 'redux'; import { Button, Confirm, @@ -28,25 +28,27 @@ import { Popup, } from 'semantic-ui-react'; -import {toMoment, serializeDate} from 'indico/utils/date'; -import {Param, Plural, PluralTranslate, Singular, Translate} from 'indico/react/i18n'; +import {ClipboardButton} from 'indico/react/components'; import {FinalCheckbox, FinalTextArea} from 'indico/react/forms'; +import {Param, Plural, PluralTranslate, Singular, Translate} from 'indico/react/i18n'; import {Responsive} from 'indico/react/util'; -import {ClipboardButton} from 'indico/react/components'; -import {DailyTimelineContent, TimelineLegend} from '../timeline'; +import {toMoment, serializeDate} from 'indico/utils/date'; + +import {openModal} from '../../actions'; +import RoomBasicDetails from '../../components/RoomBasicDetails'; +import RoomKeyLocation from '../../components/RoomKeyLocation'; +import TimeInformation from '../../components/TimeInformation'; import { getRecurrenceInfo, PopupParam, getOccurrenceTypes, transformToLegendLabels, } from '../../util'; -import RoomBasicDetails from '../../components/RoomBasicDetails'; -import RoomKeyLocation from '../../components/RoomKeyLocation'; -import TimeInformation from '../../components/TimeInformation'; -import {openModal} from '../../actions'; +import {DailyTimelineContent, TimelineLegend} from '../timeline'; + +import * as bookingsActions from './actions'; import LazyBookingObjectLink from './LazyBookingObjectLink'; import * as bookingsSelectors from './selectors'; -import * as bookingsActions from './actions'; import './BookingDetails.module.scss'; @@ -569,7 +571,13 @@ class BookingDetails extends React.Component { return transformToLegendLabels(occurrenceTypes, inactiveTypes); }; - renderActionButtons = (canCancel, canReject, showAccept, occurrenceCount, isAccepted) => { + renderActionButtons = ( + canCancel, + canReject, + showAccept, + cancellableOccurrenceCount, + isAccepted + ) => { const {bookingStateChangeInProgress} = this.props; const {actionInProgress, activeConfirmation, acceptanceFormVisible} = this.state; const rejectButton = ( @@ -609,7 +617,7 @@ class BookingDetails extends React.Component { rows={2} required /> - {isAccepted && occurrenceCount > 1 && ( + {isAccepted && cancellableOccurrenceCount > 1 && (

        - + Are you sure you want to cancel this booking? - Are you sure you want to cancel this booking? This will cancel all{' '} - } />{' '} - occurrences. + Are you sure you want to cancel this booking? This will cancel{' '} + } />{' '} + upcoming occurrences.

        -

        - {occurrenceCount > 1 && ( + {cancellableOccurrenceCount > 1 && ( +

        Single occurrences can be cancelled via the timeline view. - )} -

        +

        + )}         + ); + return ( +
        +
        +
        +
        + Favourite Users +
        +
        +
        + {loading ? ( + + ) : favoriteUsers !== null && Object.keys(favoriteUsers).length > 0 ? ( + + {Object.values(_.orderBy(favoriteUsers, ['name'])).map(user => ( + +
        +
        + {user.name} +
        + + {user.detail} + +
        + deleteFavoriteUser(user.userId)} + link + /> + } + content={Translate.string('Remove from favourites')} + position="bottom center" + /> +
        +         + ))} +         + ) : ( +
        + You have not marked any user as favourite. +
        + )} +
        +
        + u.identifier)} + onAddItems={e => e.forEach(u => addFavoriteUser(u.userId))} + triggerFactory={searchTrigger} + /> +
        + ); +} + +FavoriteUserManager.propTypes = { + userId: PropTypes.number, +}; + +FavoriteUserManager.defaultProps = { + userId: null, +}; diff --git a/indico/modules/users/client/js/Favorites.module.scss b/indico/modules/users/client/js/Favorites.module.scss new file mode 100644 index 00000000000..7451e4dad7a --- /dev/null +++ b/indico/modules/users/client/js/Favorites.module.scss @@ -0,0 +1,47 @@ +// This file is part of Indico. +// Copyright (C) 2002 - 2021 CERN +// +// Indico is free software; you can redistribute it and/or +// modify it under the terms of the MIT License; see the +// LICENSE file for more details. + +@import 'base/palette'; + +:global(.ui.button).submit-button { + margin-top: 15px; +} + +:global(.ui.list).fav-list { + :global(.item).fav-item { + margin: 0 -10px; + + .list-flex { + display: flex; + justify-content: space-between; + margin: 7px 0; + } + + .detail { + margin-top: 7px; + width: 350px; + display: inline-block; + overflow: hidden; + white-space: nowrap; + text-overflow: ellipsis; + color: $dark-gray; + } + + .delete-button { + align-self: center; + } + } +} + +.empty-favorites { + margin: 5px; +} + +:global(.ui.inline.loader.active).fav-loader { + margin: 10px 50%; + width: auto; +} diff --git a/indico/modules/users/client/js/ICSCalendarLink.jsx b/indico/modules/users/client/js/ICSCalendarLink.jsx deleted file mode 100644 index 3b07924d402..00000000000 --- a/indico/modules/users/client/js/ICSCalendarLink.jsx +++ /dev/null @@ -1,142 +0,0 @@ -// This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN -// -// Indico is free software; you can redistribute it and/or -// modify it under the terms of the MIT License; see the -// LICENSE file for more details. - -import signURL from 'indico-url:core.sign_url'; - -import React, {useState} from 'react'; -import PropTypes from 'prop-types'; -import {Button, Dropdown, Icon, Input, Label, Grid, Popup, Header} from 'semantic-ui-react'; - -import {Translate} from 'indico/react/i18n'; -import {indicoAxios, handleAxiosError} from 'indico/utils/axios'; -import {snakifyKeys} from 'indico/utils/case'; - -export default function ICSCalendarLink({endpoint, urlParams, options, ...restProps}) { - const [copied, setCopied] = useState(false); - const [option, setOption] = useState(null); - const [open, setOpen] = useState(false); - - const copyButton = ( - - } - pointing="top right" - > - - - Synchronise with your calendar - - - {options.map(({key, text, queryParams}) => ( - { - setOption({ - text, - url: await fetchURL(queryParams), - }); - setOpen(true); - }} - /> - ))} - - - ); - - return ( - { - setCopied(false); - }} - onClose={() => { - setOpen(false); - }} - wide - > -
        {option.text}} - /> - -

        - - You may copy-paste the following URL into your scheduling application. Contents will be - automatically synchronised. - -

        - - {copied && ( - - - - - - )} -         - - ); -} - -ICSCalendarLink.propTypes = { - endpoint: PropTypes.string.isRequired, - urlParams: PropTypes.objectOf(PropTypes.string), - options: PropTypes.arrayOf( - PropTypes.shape({ - key: PropTypes.string.isRequired, - text: PropTypes.string.isRequired, - queryParams: PropTypes.objectOf(PropTypes.string), - }) - ), -}; - -ICSCalendarLink.defaultProps = { - urlParams: {}, - options: [], -}; diff --git a/indico/modules/users/client/js/ProfilePicture.jsx b/indico/modules/users/client/js/ProfilePicture.jsx index 2f81edca3c4..f70403006f9 100644 --- a/indico/modules/users/client/js/ProfilePicture.jsx +++ b/indico/modules/users/client/js/ProfilePicture.jsx @@ -1,25 +1,25 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the // LICENSE file for more details. -import saveURL from 'indico-url:users.save_profile_picture'; import previewURL from 'indico-url:users.profile_picture_preview'; +import saveURL from 'indico-url:users.save_profile_picture'; +import createDecorator from 'final-form-calculate'; +import PropTypes from 'prop-types'; import React, {useState, useCallback} from 'react'; import ReactDOM from 'react-dom'; -import PropTypes from 'prop-types'; -import {Form as FinalForm, useField, useFormState} from 'react-final-form'; import {useDropzone} from 'react-dropzone'; +import {Form as FinalForm, useField, useFormState} from 'react-final-form'; import {Button, Form, Icon, Image, Card} from 'semantic-ui-react'; -import createDecorator from 'final-form-calculate'; -import {FinalSubmitButton} from 'indico/react/forms'; -import {TooltipIfTruncated} from 'indico/react/components'; -import {indicoAxios, handleAxiosError} from 'indico/utils/axios'; +import {TooltipIfTruncated} from 'indico/react/components'; +import {FinalSubmitButton} from 'indico/react/forms'; import {Translate, Param} from 'indico/react/i18n'; +import {indicoAxios, handleAxiosError} from 'indico/utils/axios'; import './ProfilePicture.module.scss'; diff --git a/indico/modules/users/client/js/ProfilePicture.module.scss b/indico/modules/users/client/js/ProfilePicture.module.scss index d2945450287..045a7cd7c08 100644 --- a/indico/modules/users/client/js/ProfilePicture.module.scss +++ b/indico/modules/users/client/js/ProfilePicture.module.scss @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/users/client/js/dashboard.jsx b/indico/modules/users/client/js/dashboard.jsx index 483a2b311c5..d539695ad98 100644 --- a/indico/modules/users/client/js/dashboard.jsx +++ b/indico/modules/users/client/js/dashboard.jsx @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the @@ -8,24 +8,21 @@ import React from 'react'; import ReactDOM from 'react-dom'; +import {ICSCalendarLink} from 'indico/react/components'; import {Translate} from 'indico/react/i18n'; -import ICSCalendarLink from './ICSCalendarLink'; - document.addEventListener('DOMContentLoaded', () => { - const userId = document.querySelector('body').dataset.userId; ReactDOM.render( , document.querySelector('#dashboard-calendar-link') diff --git a/indico/modules/users/client/js/index.js b/indico/modules/users/client/js/index.js index 0a52ca2a3da..848cf1c41a0 100644 --- a/indico/modules/users/client/js/index.js +++ b/indico/modules/users/client/js/index.js @@ -1,5 +1,5 @@ // This file is part of Indico. -// Copyright (C) 2002 - 2020 CERN +// Copyright (C) 2002 - 2021 CERN // // Indico is free software; you can redistribute it and/or // modify it under the terms of the MIT License; see the diff --git a/indico/modules/users/controllers.py b/indico/modules/users/controllers.py index 33b1d570b1c..b22ac7d8662 100644 --- a/indico/modules/users/controllers.py +++ b/indico/modules/users/controllers.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from collections import namedtuple from io import BytesIO from operator import attrgetter, itemgetter @@ -17,20 +15,19 @@ from marshmallow import fields from marshmallow_enum import EnumField from PIL import Image -from sqlalchemy.orm import joinedload, load_only, subqueryload, undefer +from sqlalchemy.orm import joinedload, load_only, subqueryload from sqlalchemy.orm.exc import StaleDataError from webargs import validate from werkzeug.exceptions import BadRequest, Forbidden, NotFound -from werkzeug.http import parse_date from indico.core import signals from indico.core.auth import multipass +from indico.core.cache import make_scoped_cache from indico.core.db import db from indico.core.db.sqlalchemy.util.queries import get_n_matching from indico.core.errors import UserValueError from indico.core.marshmallow import mm from indico.core.notifications import make_email, send_email -from indico.legacy.common.cache import GenericCache from indico.modules.admin import RHAdminBase from indico.modules.auth import Identity from indico.modules.auth.models.registration_requests import RegistrationRequest @@ -44,12 +41,12 @@ from indico.modules.users.models.emails import UserEmail from indico.modules.users.models.users import ProfilePictureSource from indico.modules.users.operations import create_user +from indico.modules.users.schemas import BasicCategorySchema from indico.modules.users.util import (get_gravatar_for_user, get_linked_events, get_related_categories, - get_suggested_categories, merge_users, search_users, serialize_user, + get_suggested_categories, merge_users, search_users, send_avatar, serialize_user, set_user_avatar) -from indico.modules.users.views import WPUser, WPUserDashboard, WPUserProfilePic, WPUsersAdmin +from indico.modules.users.views import WPUser, WPUserDashboard, WPUserFavorites, WPUserProfilePic, WPUsersAdmin from indico.util.date_time import now_utc -from indico.util.event import truncate_path from indico.util.i18n import _ from indico.util.images import square from indico.util.marshmallow import HumanizedDate, Principal, validate_with_message @@ -60,12 +57,12 @@ from indico.web.flask.util import send_file, url_for from indico.web.forms.base import FormDefaults from indico.web.http_api.metadata import Serializer -from indico.web.rh import RHProtected, RHTokenProtected -from indico.web.util import jsonify_data, jsonify_form, jsonify_template +from indico.web.rh import RH, RHProtected, allow_signed_url +from indico.web.util import is_legacy_signed_url_valid, jsonify_data, jsonify_form, jsonify_template IDENTITY_ATTRIBUTES = {'first_name', 'last_name', 'email', 'affiliation', 'full_name'} -UserEntry = namedtuple('UserEntry', IDENTITY_ATTRIBUTES | {'profile_url', 'user'}) +UserEntry = namedtuple('UserEntry', IDENTITY_ATTRIBUTES | {'profile_url', 'avatar_url', 'user'}) def get_events_in_categories(category_ids, user, limit=10): @@ -133,13 +130,13 @@ def _process(self): categories = get_related_categories(self.user) categories_events = [] if categories: - category_ids = {c['categ'].id for c in categories.itervalues()} + category_ids = {c['categ'].id for c in categories.values()} categories_events = get_events_in_categories(category_ids, self.user) from_dt = now_utc(False) - relativedelta(weeks=1, hour=0, minute=0, second=0) linked_events = [(event, {'management': bool(roles & self.management_roles), 'reviewing': bool(roles & self.reviewer_roles), 'attendance': bool(roles & self.attendance_roles)}) - for event, roles in get_linked_events(self.user, from_dt, 10).iteritems()] + for event, roles in get_linked_events(self.user, from_dt, 10).items()] return WPUserDashboard.render_template('dashboard.html', 'dashboard', user=self.user, categories=categories, @@ -148,31 +145,32 @@ def _process(self): linked_events=linked_events) -class RHExportDashboardICS(RHTokenProtected): +@allow_signed_url +class RHExportDashboardICS(RHProtected): + def _get_user(self): + return session.user + @use_kwargs({ 'from_': HumanizedDate(data_key='from', missing=lambda: now_utc(False) - relativedelta(weeks=1)), 'include': fields.List(fields.Str(), missing={'linked', 'categories'}), 'limit': fields.Integer(missing=100, validate=lambda v: 0 < v <= 500) - }) + }, location='query') def _process(self, from_, include, limit): - categories = get_related_categories(self.user) - categories_events = [] - if categories: - category_ids = {c['categ'].id for c in categories.itervalues()} - categories_events = get_events_in_categories(category_ids, self.user, limit=limit) - - linked_events = get_linked_events( - self.user, - from_, - limit=limit, - load_also=('description', 'own_room_id', 'own_venue_id', 'own_room_name', 'own_venue_name') - ) - + user = self._get_user() all_events = set() + if 'linked' in include: - all_events |= set(linked_events) - if 'categories' in include: - all_events |= set(categories_events) + all_events |= set(get_linked_events( + user, + from_, + limit=limit, + load_also=('description', 'own_room_id', 'own_venue_id', 'own_room_name', 'own_venue_name') + )) + + if 'categories' in include and (categories := get_related_categories(user)): + category_ids = {c['categ'].id for c in categories.values()} + all_events |= set(get_events_in_categories(category_ids, user, limit=limit)) + all_events = sorted(all_events, key=lambda e: (e.start_dt, e.id))[:limit] response = {'results': [serialize_event_for_ical(event, 'events') for event in all_events]} @@ -180,6 +178,20 @@ def _process(self, from_, include, limit): return send_file('event.ics', BytesIO(serializer(response)), 'text/calendar') +class RHExportDashboardICSLegacy(RHExportDashboardICS): + def _get_user(self): + user = User.get_or_404(request.view_args['user_id'], is_deleted=False) + if not is_legacy_signed_url_valid(user, request.full_path): + raise BadRequest('Invalid signature') + if user.is_blocked: + raise BadRequest('User blocked') + return user + + def _check_access(self): + # disable the usual RHProtected access check; `_get_user` does it all + pass + + class RHPersonalData(RHUserBase): allow_system_user = True @@ -209,14 +221,12 @@ class RHProfilePicturePreview(RHUserBase): This always uses a fresh picture without any caching. """ - @use_kwargs({ - 'source': EnumField(ProfilePictureSource, location='view_args') - }) + @use_kwargs({'source': EnumField(ProfilePictureSource)}, location='view_args') def _process(self, source): if source == ProfilePictureSource.standard: first_name = self.user.first_name[0].upper() if self.user.first_name else '' avatar = render_template('users/avatar.svg', bg_color=self.user.avatar_bg_color, text=first_name) - return send_file('avatar.svg', BytesIO(avatar.encode('utf-8')), mimetype='image/svg+xml', + return send_file('avatar.svg', BytesIO(avatar.encode()), mimetype='image/svg+xml', no_cache=True, inline=True, safe=False) elif source == ProfilePictureSource.custom: metadata = self.user.picture_metadata @@ -227,22 +237,14 @@ def _process(self, source): return send_file('avatar.png', BytesIO(gravatar), mimetype='image/png') -class RHProfilePictureDisplay(RHUserBase): +class RHProfilePictureDisplay(RH): """Display the user's profile picture.""" - allow_system_user = True + def _process_args(self): + self.user = User.get_or_404(request.view_args['user_id'], is_deleted=False) def _process(self): - if self.user.picture_source == ProfilePictureSource.standard: - first_name = self.user.first_name[0].upper() if self.user.first_name else '' - avatar = render_template('users/avatar.svg', bg_color=self.user.avatar_bg_color, text=first_name) - return send_file('avatar.svg', BytesIO(avatar.encode('utf-8')), mimetype='image/svg+xml', - no_cache=False, inline=True, safe=False, cache_timeout=(86400*7)) - - metadata = self.user.picture_metadata - return send_file('avatar.png', BytesIO(self.user.picture), mimetype=metadata['content_type'], - inline=True, conditional=True, last_modified=parse_date(metadata['lastmod']), - cache_timeout=(86400*7)) + return send_avatar(self.user) class RHSaveProfilePicture(RHUserBase): @@ -264,7 +266,7 @@ def _process(self, source): f = request.files['picture'] try: pic = Image.open(f) - except IOError: + except OSError: raise UserValueError(_('You cannot upload this file as profile picture.')) if pic.format.lower() not in {'jpeg', 'png', 'gif', 'webp'}: raise UserValueError(_('The file has an invalid format ({format}).').format(format=pic.format)) @@ -310,40 +312,42 @@ def _process(self): class RHUserFavorites(RHUserBase): def _process(self): - query = (Category.query - .filter(Category.id.in_(c.id for c in self.user.favorite_categories)) - .options(undefer('chain_titles'))) - categories = sorted([(cat, truncate_path(cat.chain_titles[:-1], chars=50)) for cat in query], - key=lambda c: (c[0].title, c[1])) - return WPUser.render_template('favorites.html', 'favorites', user=self.user, favorite_categories=categories) + return WPUserFavorites.render_template('favorites.html', 'favorites', user=self.user) -class RHUserFavoritesUsersAdd(RHUserBase): - def _process(self): - users = [User.get(int(id_)) for id_ in request.form.getlist('user_id')] - self.user.favorite_users |= set(filter(None, users)) - tpl = get_template_module('users/_favorites.html') - return jsonify(success=True, users=[serialize_user(user) for user in users], - html=tpl.favorite_users_list(self.user)) +class RHUserFavoritesAPI(RHUserBase): + def _process_args(self): + RHUserBase._process_args(self) + self.fav_user = ( + User.get_or_404(request.view_args['fav_user_id']) if 'fav_user_id' in request.view_args else None + ) + def _process_GET(self): + return jsonify(sorted(u.id for u in self.user.favorite_users)) -class RHUserFavoritesUserRemove(RHUserBase): - def _process(self): - user = User.get(int(request.view_args['fav_user_id'])) - self.user.favorite_users.discard(user) - try: - db.session.flush() - except StaleDataError: - # Deleted in another transaction - db.session.rollback() - return jsonify(success=True) + def _process_PUT(self): + self.user.favorite_users.add(self.fav_user) + return jsonify(self.user.id), 201 + + def _process_DELETE(self): + self.user.favorite_users.discard(self.fav_user) + return '', 204 class RHUserFavoritesCategoryAPI(RHUserBase): def _process_args(self): RHUserBase._process_args(self) - self.category = Category.get_or_404(request.view_args['category_id']) - self.suggestion = self.user.suggested_categories.filter_by(category=self.category).first() + self.category = ( + Category.get_or_404(request.view_args['category_id']) if 'category_id' in request.view_args else None + ) + self.suggestion = ( + self.user.suggested_categories.filter_by(category=self.category).first() + if 'category_id' in request.view_args + else None + ) + + def _process_GET(self): + return jsonify({d.id: BasicCategorySchema().dump(d) for d in self.user.favorite_categories}) def _process_PUT(self): if self.category not in self.user.favorite_categories: @@ -378,10 +382,10 @@ def _process(self): class RHUserEmails(RHUserBase): def _send_confirmation(self, email): - token_storage = GenericCache('confirm-email') + token_storage = make_scoped_cache('confirm-email') data = {'email': email, 'user_id': self.user.id} token = make_unique_token(lambda t: not token_storage.get(t)) - token_storage.set(token, data, 24 * 3600) + token_storage.set(token, data, timeout=86400) send_email(make_email(email, template=get_template_module('users/emails/verify_email.txt', user=self.user, email=email, token=token))) @@ -397,7 +401,7 @@ def _process(self): class RHUserEmailsVerify(RHUserBase): flash_user_status = False - token_storage = GenericCache('confirm-email') + token_storage = make_scoped_cache('confirm-email') def _validate(self, data): if not data: @@ -407,7 +411,7 @@ def _validate(self, data): if not user or user != self.user: flash(_('This token is for a different Indico user. Please login with the correct account'), 'error') return False, None - existing = UserEmail.find_first(is_user_deleted=False, email=data['email']) + existing = UserEmail.query.filter_by(is_user_deleted=False, email=data['email']).first() if existing and not existing.user.is_pending: if existing.user == self.user: flash(_('This email address is already attached to your account.')) @@ -464,7 +468,7 @@ def _process(self): class RHAdmins(RHAdminBase): - """Show Indico administrators""" + """Show Indico administrators.""" def _process(self): admins = set(User.query @@ -489,14 +493,14 @@ def _process(self): class RHUsersAdmin(RHAdminBase): - """Admin users overview""" + """Admin users overview.""" def _process(self): form = SearchForm(obj=FormDefaults(exact=True)) form_data = form.data search_results = None num_of_users = User.query.count() - num_deleted_users = User.find(is_deleted=True).count() + num_deleted_users = User.query.filter_by(is_deleted=True).count() if form.validate_on_submit(): search_results = [] @@ -504,24 +508,32 @@ def _process(self): include_deleted = form_data.pop('include_deleted') include_pending = form_data.pop('include_pending') external = form_data.pop('external') - form_data = {k: v for (k, v) in form_data.iteritems() if v and v.strip()} + form_data = {k: v for (k, v) in form_data.items() if v and v.strip()} matches = search_users(exact=exact, include_deleted=include_deleted, include_pending=include_pending, include_blocked=True, external=external, allow_system_user=True, **form_data) for entry in matches: if isinstance(entry, User): search_results.append(UserEntry( + avatar_url=entry.avatar_url, profile_url=url_for('.user_profile', entry), user=entry, **{k: getattr(entry, k) for k in IDENTITY_ATTRIBUTES} )) else: + if not entry.data['first_name'] and not entry.data['last_name']: + full_name = '' + initial = '?' + else: + full_name = f'{entry.data["first_name"]} {entry.data["last_name"]}'.strip() + initial = full_name[0] search_results.append(UserEntry( + avatar_url=url_for('assets.avatar', name=initial), profile_url=None, user=None, - full_name="{first_name} {last_name}".format(**entry.data.to_dict()), + full_name=full_name, **{k: entry.data.get(k) for k in (IDENTITY_ATTRIBUTES - {'full_name'})} )) - search_results.sort(key=attrgetter('first_name', 'last_name')) + search_results.sort(key=attrgetter('full_name')) num_reg_requests = RegistrationRequest.query.count() return WPUsersAdmin.render_template('users_admin.html', 'users', form=form, search_results=search_results, @@ -541,7 +553,7 @@ def _process(self): class RHUsersAdminCreate(RHAdminBase): - """Create user (admin)""" + """Create user (admin).""" def _process(self): form = AdminAccountRegistrationForm() @@ -594,7 +606,7 @@ def _get_merge_problems(source, target): class RHUsersAdminMerge(RHAdminBase): - """Merge users (admin)""" + """Merge users (admin).""" def _process(self): form = MergeForm() @@ -620,14 +632,14 @@ class RHUsersAdminMergeCheck(RHAdminBase): @use_kwargs({ 'source': Principal(allow_external_users=True, required=True), 'target': Principal(allow_external_users=True, required=True), - }) + }, location='query') def _process(self, source, target): errors, warnings = _get_merge_problems(source, target) return jsonify(errors=errors, warnings=warnings, source=serialize_user(source), target=serialize_user(target)) class RHRegistrationRequestList(RHAdminBase): - """List all registration requests""" + """List all registration requests.""" def _process(self): requests = RegistrationRequest.query.order_by(RegistrationRequest.email).all() @@ -635,7 +647,7 @@ def _process(self): class RHRegistrationRequestBase(RHAdminBase): - """Base class to process a registration request""" + """Base class to process a registration request.""" def _process_args(self): RHAdminBase._process_args(self) @@ -643,7 +655,7 @@ def _process_args(self): class RHAcceptRegistrationRequest(RHRegistrationRequestBase): - """Accept a registration request""" + """Accept a registration request.""" def _process(self): user, identity = register_user(self.request.email, self.request.extra_emails, self.request.user_data, @@ -655,7 +667,7 @@ def _process(self): class RHRejectRegistrationRequest(RHRegistrationRequestBase): - """Reject a registration request""" + """Reject a registration request.""" def _process(self): db.session.delete(self.request) @@ -665,25 +677,25 @@ def _process(self): return jsonify_data() -class UserSearchResultSchema(mm.ModelSchema): +class UserSearchResultSchema(mm.SQLAlchemyAutoSchema): class Meta: model = User - fields = ('id', 'identifier', 'email', 'affiliation', 'full_name') + fields = ('id', 'identifier', 'email', 'affiliation', 'full_name', 'first_name', 'last_name') search_result_schema = UserSearchResultSchema() class RHUserSearch(RHProtected): - """Search for users based on given criteria""" + """Search for users based on given criteria.""" def _serialize_pending_user(self, entry): first_name = entry.data.get('first_name') or '' last_name = entry.data.get('last_name') or '' - full_name = '{} {}'.format(first_name, last_name).strip() or 'Unknown' + full_name = f'{first_name} {last_name}'.strip() or 'Unknown' affiliation = entry.data.get('affiliation') or '' email = entry.data['email'].lower() - ext_id = '{}:{}'.format(entry.provider.name, entry.identifier) + ext_id = f'{entry.provider.name}:{entry.identifier}' # detailed data to put in redis to create a pending user if needed self.externals[ext_id] = { 'first_name': first_name, @@ -697,10 +709,12 @@ def _serialize_pending_user(self, entry): return { '_ext_id': ext_id, 'id': None, - 'identifier': 'ExternalUser:{}'.format(ext_id), + 'identifier': f'ExternalUser:{ext_id}', 'email': email, 'affiliation': affiliation, 'full_name': full_name, + 'first_name': first_name, + 'last_name': last_name, } def _serialize_entry(self, entry): @@ -710,24 +724,24 @@ def _serialize_entry(self, entry): return self._serialize_pending_user(entry) def _process_pending_users(self, results): - cache = GenericCache('external-user') + cache = make_scoped_cache('external-user') for entry in results: ext_id = entry.pop('_ext_id', None) if ext_id is not None: - cache.set(ext_id, self.externals[ext_id], 86400) + cache.set(ext_id, self.externals[ext_id], timeout=86400) @use_kwargs({ 'first_name': fields.Str(validate=validate.Length(min=1)), 'last_name': fields.Str(validate=validate.Length(min=1)), - 'email': fields.Str(validate=lambda s: len(s) > 3 and '@' in s), + 'email': fields.Str(validate=lambda s: len(s) > 3), 'affiliation': fields.Str(validate=validate.Length(min=1)), 'exact': fields.Bool(missing=False), 'external': fields.Bool(missing=False), 'favorites_first': fields.Bool(missing=False) }, validate=validate_with_message( - lambda args: args.viewkeys() & {'first_name', 'last_name', 'email', 'affiliation'}, + lambda args: args.keys() & {'first_name', 'last_name', 'email', 'affiliation'}, 'No criteria provided' - )) + ), location='query') def _process(self, exact, external, favorites_first, **criteria): matches = search_users(exact=exact, include_pending=True, external=external, **criteria) self.externals = {} @@ -743,7 +757,7 @@ def _process(self, exact, external, favorites_first, **criteria): class RHUserSearchInfo(RHProtected): def _process(self): - external_users_available = any(auth.supports_search for auth in multipass.identity_providers.itervalues()) + external_users_available = any(auth.supports_search for auth in multipass.identity_providers.values()) return jsonify(external_users_available=external_users_available) diff --git a/indico/modules/users/ext.py b/indico/modules/users/ext.py index 5b673de6c15..917c013098d 100644 --- a/indico/modules/users/ext.py +++ b/indico/modules/users/ext.py @@ -1,15 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - - -class ExtraUserPreferences(object): - """Defines additional user preferences +class ExtraUserPreferences: + """Define additional user preferences. To use this class, subclass it and override `defaults`, `fields` and `save` to implement your custom logic. @@ -39,11 +36,11 @@ def save(self, data): # to be called/used when implementing custom settings. def extend_defaults(self, defaults): - """Adds values to the FormDefaults.""" - for key, value in self.load().iteritems(): + """Add values to the FormDefaults.""" + for key, value in self.load().items(): key = self._prefix + key if hasattr(defaults, key): - raise RuntimeError('Preference collision: {}'.format(key)) + raise RuntimeError(f'Preference collision: {key}') defaults[key] = value def process_form_data(self, data): @@ -58,11 +55,11 @@ def process_form_data(self, data): self.save(local_data) def extend_form(self, form_class): - """Create a subclass of the form containing the extra field""" - form_class = type(b'ExtendedUserPreferencesForm', (form_class,), {}) - for name, field in self.fields.iteritems(): + """Create a subclass of the form containing the extra field.""" + form_class = type('ExtendedUserPreferencesForm', (form_class,), {}) + for name, field in self.fields.items(): name = self._prefix + name if hasattr(form_class, name): - raise RuntimeError('Preference collision: {}'.format(name)) + raise RuntimeError(f'Preference collision: {name}') setattr(form_class, name, field) return form_class diff --git a/indico/modules/users/forms.py b/indico/modules/users/forms.py index 523d9ad99c0..10b85c6cda4 100644 --- a/indico/modules/users/forms.py +++ b/indico/modules/users/forms.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from operator import itemgetter from pytz import common_timezones, common_timezones_set @@ -29,7 +27,7 @@ class UserDetailsForm(SyncedInputsMixin, IndicoForm): - title = IndicoEnumSelectField(_('Title'), enum=UserTitle) + title = IndicoEnumSelectField(_('Title'), enum=UserTitle, sorted=True) first_name = StringField(_('First name'), [used_if_not_synced, DataRequired()], widget=SyncedInputWidget()) last_name = StringField(_('Family name'), [used_if_not_synced, DataRequired()], widget=SyncedInputWidget()) affiliation = StringField(_('Affiliation'), widget=SyncedInputWidget()) @@ -65,12 +63,12 @@ class UserPreferencesForm(IndicoForm): description=_('The previewer is used by default for image and text files, but not for PDF files.')) def __init__(self, *args, **kwargs): - super(UserPreferencesForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) - locales = [(code, '{} ({})'.format(name, territory) if territory else name) - for code, (name, territory) in get_all_locales().iteritems()] + locales = [(code, f'{name} ({territory})' if territory else name) + for code, (name, territory) in get_all_locales().items()] self.lang.choices = sorted(locales, key=itemgetter(1)) - self.timezone.choices = zip(common_timezones, common_timezones) + self.timezone.choices = list(zip(common_timezones, common_timezones)) if self.timezone.object_data and self.timezone.object_data not in common_timezones_set: self.timezone.choices.append((self.timezone.object_data, self.timezone.object_data)) @@ -79,7 +77,13 @@ class UserEmailsForm(IndicoForm): email = EmailField(_('Add new email address'), [DataRequired(), Email()], filters=[lambda x: x.lower() if x else x]) def validate_email(self, field): - if UserEmail.find(~User.is_pending, is_user_deleted=False, email=field.data, _join=User).count(): + conflict = (UserEmail.query + .filter(~User.is_pending, + ~UserEmail.is_user_deleted, + UserEmail.email == field.data) + .join(User) + .has_rows()) + if conflict: raise ValidationError(_('This email address is already in use.')) @@ -116,7 +120,7 @@ def __init__(self, *args, **kwargs): if config.LOCAL_IDENTITIES: for field in ('username', 'password', 'confirm_password'): inject_validators(self, field, [HiddenUnless('create_identity')], early=True) - super(AdminAccountRegistrationForm, self).__init__(*args, **kwargs) + super().__init__(*args, **kwargs) del self.comment if not config.LOCAL_IDENTITIES: del self.username diff --git a/indico/modules/users/legacy.py b/indico/modules/users/legacy.py deleted file mode 100644 index f9cd1f044db..00000000000 --- a/indico/modules/users/legacy.py +++ /dev/null @@ -1,335 +0,0 @@ -# This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN -# -# Indico is free software; you can redistribute it and/or -# modify it under the terms of the MIT License; see the -# LICENSE file for more details. - -from flask_multipass import IdentityInfo - -from indico.legacy.common.cache import GenericCache -from indico.legacy.fossils.user import IAvatarFossil, IAvatarMinimalFossil -from indico.modules.auth import Identity -from indico.modules.users import User, logger -from indico.util.caching import memoize_request -from indico.util.fossilize import Fossilizable, fossilizes -from indico.util.locators import locator_property -from indico.util.string import encode_utf8, return_ascii, to_unicode - - -AVATAR_FIELD_MAP = { - 'email': 'email', - 'name': 'first_name', - 'surName': 'last_name', - 'organisation': 'affiliation' -} - - -class AvatarUserWrapper(Fossilizable): - """Avatar-like wrapper class that holds a DB-stored user.""" - - fossilizes(IAvatarFossil, IAvatarMinimalFossil) - - def __init__(self, user_id): - self.id = str(user_id) - - @property - @memoize_request - def _original_user(self): - # A proper user, with an id that can be mapped directly to sqlalchemy - if isinstance(self.id, int) or self.id.isdigit(): - return User.get(int(self.id)) - # A user who had no real indico account but an ldap identifier/email. - # In this case we try to find his real user and replace the ID of this object - # with that user's ID. - data = self.id.split(':') - # TODO: Once everything is in SQLAlchemy this whole thing needs to go away! - user = None - if data[0] == 'LDAP': - identifier = data[1] - email = data[2] - # You better have only one ldap provider or at least different identifiers ;) - identity = Identity.query.filter(Identity.provider != 'indico', Identity.identifier == identifier).first() - if identity: - user = identity.user - elif data[0] == 'Nice': - email = data[1] - else: - return None - if not user: - user = User.query.filter(User.all_emails == email).first() - if user: - self._old_id = self.id - self.id = str(user.id) - logger.info("Updated legacy user id (%s => %s)", self._old_id, self.id) - return user - - @property - @memoize_request - def user(self): - user = self._original_user - if user is not None and user.is_deleted and user.merged_into_id is not None: - while user.merged_into_id is not None: - user = user.merged_into_user - return user - - def getId(self): - return str(self.user.id) if self.user else str(self.id) - - @property - def api_key(self): - return self.user.api_key if self.user else None - - def getStatus(self): - return 'deleted' if not self.user or self.user.is_deleted else 'activated' - - def isActivated(self): - # All accounts are activated during the transition period - return True - - def isDisabled(self): - # The user has been blocked or deleted (due to merge) - return not self.user or self.user.is_blocked or self.user.is_deleted - - def setName(self, name, reindex=False): - self.user.first_name = to_unicode(name) - - @encode_utf8 - def getName(self): - return self.user.first_name if self.user else '' - - getFirstName = getName - - def setSurName(self, surname, reindex=False): - self.user.last_name = to_unicode(surname) - - @encode_utf8 - def getSurName(self): - return self.user.last_name if self.user else '' - - getFamilyName = getSurName - - @encode_utf8 - def getFullName(self): - if not self.user: - return '' - return self.user.get_full_name(last_name_first=True, last_name_upper=True, - abbrev_first_name=False, show_title=False) - - @encode_utf8 - def getStraightFullName(self, upper=True): - if not self.user: - return '' - return self.user.get_full_name(last_name_first=False, last_name_upper=upper, - abbrev_first_name=False, show_title=False) - - getDirectFullNameNoTitle = getStraightFullName - - @encode_utf8 - def getAbrName(self): - if not self.user: - return '' - return self.user.get_full_name(last_name_first=True, last_name_upper=False, - abbrev_first_name=True, show_title=False) - - @encode_utf8 - def getStraightAbrName(self): - if not self.user: - return '' - return self.user.get_full_name(last_name_first=False, last_name_upper=False, - abbrev_first_name=True, show_title=False) - - def setOrganisation(self, affiliation, reindex=False): - self.user.affiliation = to_unicode(affiliation) - - @encode_utf8 - def getOrganisation(self): - return self.user.affiliation if self.user else '' - - getAffiliation = getOrganisation - - def setTitle(self, title): - self.user.title = to_unicode(title) - - @encode_utf8 - def getTitle(self): - return self.user.title if self.user else '' - - def setTimezone(self, tz): - self.user.settings.set('timezone', to_unicode(tz)) - - @encode_utf8 - def getAddress(self): - return self.user.address if self.user else '' - - def setAddress(self, address): - self.user.address = to_unicode(address) - - def getEmails(self): - # avoid 'stale association proxy' - user = self.user - return set(user.all_emails) if user else set() - - @encode_utf8 - def getEmail(self): - return self.user.email if self.user else '' - - email = property(getEmail) - - def setEmail(self, email, reindex=False): - self.user.email = to_unicode(email) - - def hasEmail(self, email): - user = self.user # avoid 'stale association proxy' - if not user: - return False - return email.lower() in user.all_emails - - @encode_utf8 - def getTelephone(self): - return self.user.phone if self.user else '' - - def getFax(self): - # Some older code still clones fax, etc... - # it's never shown in the interface anyway. - return '' - - getPhone = getTelephone - - def setTelephone(self, phone): - self.user.phone = to_unicode(phone) - - setPhone = setTelephone - - def canUserModify(self, avatar): - if not self.user: - return False - return avatar.id == str(self.user.id) or avatar.user.is_admin - - @locator_property - def locator(self): - d = {} - if self.user: - d['userId'] = self.user.id - return d - - def isAdmin(self): - if not self.user: - return False - return self.user.is_admin - - @property - def as_new(self): - return self.user - - def __eq__(self, other): - if not isinstance(other, (AvatarUserWrapper, User)): - return False - elif str(self.id) == str(other.id): - return True - elif self.user: - return str(self.user.id) == str(other.id) - else: - return False - - def __ne__(self, other): - return not (self == other) - - def __hash__(self): - return hash(str(self.id)) - - @return_ascii - def __repr__(self): - if self.user is None: - return u''.format(self.id) - elif self._original_user.merged_into_user: - return u''.format( - self.id, self._original_user.full_name, self._original_user.email, self.user.id) - else: - return u''.format(self.id, self.user.full_name, self.user.email) - - -class AvatarProvisionalWrapper(Fossilizable): - """ - Wraps provisional data for users that are not in the DB yet - """ - - fossilizes(IAvatarFossil, IAvatarMinimalFossil) - - def __init__(self, identity_info): - self.identity_info = identity_info - self.data = identity_info.data - - def getId(self): - return u"{}:{}".format(self.identity_info.provider.name, self.identity_info.identifier) - - id = property(getId) - - @encode_utf8 - def getEmail(self): - return self.data['email'] - - def getEmails(self): - return [self.data['email']] - - @encode_utf8 - def getFirstName(self): - return self.data.get('first_name', '') - - @encode_utf8 - def getFamilyName(self): - return self.data.get('last_name', '') - - @encode_utf8 - def getStraightFullName(self, upper=False): - last_name = to_unicode(self.data.get('last_name', '')) - if upper: - last_name = last_name.upper() - return u'{} {}'.format(to_unicode(self.data.get('first_name', '')), last_name) - - def getTitle(self): - return '' - - @encode_utf8 - def getTelephone(self): - return self.data.get('phone', '') - - getPhone = getTelephone - - @encode_utf8 - def getOrganisation(self): - return self.data.get('affiliation', '') - - getAffiliation = getOrganisation - - def getFax(self): - return None - - def getAddress(self): - return u'' - - @return_ascii - def __repr__(self): - return u''.format( - self.identity_info.provider.name, - self.identity_info.identifier, - **self.data.to_dict()) - - -def search_avatars(criteria, exact=False, search_externals=False): - from indico.modules.users.util import search_users - - if not any(criteria.viewvalues()): - return [] - - def _process_identities(obj): - if isinstance(obj, IdentityInfo): - GenericCache('pending_identities').set('{}:{}'.format(obj.provider.name, obj.identifier), obj.data) - return AvatarProvisionalWrapper(obj) - else: - return obj.as_avatar - - results = search_users(exact=exact, external=search_externals, - **{AVATAR_FIELD_MAP[k]: v for (k, v) in criteria.iteritems() if v}) - - return [_process_identities(obj) for obj in results] diff --git a/indico/modules/users/models/affiliations.py b/indico/modules/users/models/affiliations.py index 098907723af..86043a22b20 100644 --- a/indico/modules/users/models/affiliations.py +++ b/indico/modules/users/models/affiliations.py @@ -1,15 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy.custom.unaccent import define_unaccented_lowercase_index -from indico.util.string import return_ascii class UserAffiliation(db.Model): @@ -39,9 +36,8 @@ class UserAffiliation(db.Model): # relationship backrefs: # - user (User._affiliation) - @return_ascii def __repr__(self): - return ''.format(self.id, self.name, self.user) + return f'' define_unaccented_lowercase_index(UserAffiliation.name) diff --git a/indico/modules/users/models/emails.py b/indico/modules/users/models/emails.py index cf72c9c56b0..20c59ab75e7 100644 --- a/indico/modules/users/models/emails.py +++ b/indico/modules/users/models/emails.py @@ -1,15 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db from indico.core.db.sqlalchemy.custom.unaccent import define_unaccented_lowercase_index -from indico.util.string import return_ascii class UserEmail(db.Model): @@ -54,9 +51,8 @@ class UserEmail(db.Model): # relationship backrefs: # - user (User._all_emails) - @return_ascii def __repr__(self): - return ''.format(self.id, self.email, self.is_primary) + return f'' define_unaccented_lowercase_index(UserEmail.email) diff --git a/indico/modules/users/models/favorites.py b/indico/modules/users/models/favorites.py index 2bf894478f3..384012aca38 100644 --- a/indico/modules/users/models/favorites.py +++ b/indico/modules/users/models/favorites.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db diff --git a/indico/modules/users/models/settings.py b/indico/modules/users/models/settings.py index a58fbe5ae7a..f53c8b8d24a 100644 --- a/indico/modules/users/models/settings.py +++ b/indico/modules/users/models/settings.py @@ -1,23 +1,20 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from functools import wraps from indico.core.db import db from indico.core.settings import SettingsProxyBase from indico.core.settings.models.base import JSONSettingsBase from indico.core.settings.util import get_all_settings, get_setting -from indico.util.string import return_ascii class UserSetting(JSONSettingsBase, db.Model): - """User-specific settings""" + """User-specific settings.""" __table_args__ = (db.Index(None, 'user_id', 'module', 'name'), db.Index(None, 'user_id', 'module'), db.UniqueConstraint('user_id', 'module', 'name'), @@ -42,9 +39,8 @@ class UserSetting(JSONSettingsBase, db.Model): ) ) - @return_ascii def __repr__(self): - return ''.format(self.user_id, self.module, self.name, self.value) + return f'' def user_or_id(f): @@ -67,16 +63,16 @@ def wrapper(self, user, *args, **kwargs): class UserSettingsProxy(SettingsProxyBase): - """Proxy class to access user-specific settings for a certain module""" + """Proxy class to access user-specific settings for a certain module.""" @property def query(self): - """Returns a query object filtering by the proxy's module.""" - return UserSetting.find(module=self.module) + """Return a query object filtering by the proxy's module.""" + return UserSetting.query.filter_by(module=self.module) @user_or_id def get_all(self, user, no_defaults=False): - """Retrieves all settings + """Retrieve all settings. :param user: ``{'user': user}`` or ``{'user_id': id}`` :param no_defaults: Only return existing settings and ignore defaults. @@ -86,7 +82,7 @@ def get_all(self, user, no_defaults=False): @user_or_id def get(self, user, name, default=SettingsProxyBase.default_sentinel): - """Retrieves the value of a single setting. + """Retrieve the value of a single setting. :param user: ``{'user': user}`` or ``{'user_id': id}`` :param name: Setting name @@ -98,7 +94,7 @@ def get(self, user, name, default=SettingsProxyBase.default_sentinel): @user_or_id def set(self, user, name, value): - """Sets a single setting. + """Set a single setting. :param user: ``{'user': user}`` or ``{'user_id': id}`` :param name: Setting name @@ -110,20 +106,20 @@ def set(self, user, name, value): @user_or_id def set_multi(self, user, items): - """Sets multiple settings at once. + """Set multiple settings at once. :param user: ``{'user': user}`` or ``{'user_id': id}`` :param items: Dict containing the new settings """ for name in items: self._check_name(name) - items = {k: self._convert_from_python(k, v) for k, v in items.iteritems()} + items = {k: self._convert_from_python(k, v) for k, v in items.items()} UserSetting.set_multi(self.module, items, **user) self._flush_cache() @user_or_id def delete(self, user, *names): - """Deletes settings. + """Delete settings. :param user: ``{'user': user}`` or ``{'user_id': id}`` :param names: One or more names of settings to delete @@ -135,7 +131,7 @@ def delete(self, user, *names): @user_or_id def delete_all(self, user): - """Deletes all settings. + """Delete all settings. :param user: ``{'user': user}`` or ``{'user_id': id}`` """ diff --git a/indico/modules/users/models/suggestions.py b/indico/modules/users/models/suggestions.py index f2ab425e9c8..b17c78d5926 100644 --- a/indico/modules/users/models/suggestions.py +++ b/indico/modules/users/models/suggestions.py @@ -1,14 +1,12 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core.db import db -from indico.util.string import format_repr, return_ascii +from indico.util.string import format_repr class SuggestedCategory(db.Model): @@ -53,7 +51,6 @@ class SuggestedCategory(db.Model): # relationship backrefs: # - user (User.suggested_categories) - @return_ascii def __repr__(self): return format_repr(self, 'user_id', 'category_id', 'score', is_ignored=False) diff --git a/indico/modules/users/models/users.py b/indico/modules/users/models/users.py index 13758f455dc..b5c7f4ad607 100644 --- a/indico/modules/users/models/users.py +++ b/indico/modules/users/models/users.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - import itertools from enum import Enum from operator import attrgetter @@ -21,6 +19,7 @@ from sqlalchemy.orm import object_session from werkzeug.utils import cached_property +from indico.core import signals from indico.core.auth import multipass from indico.core.db import db from indico.core.db.sqlalchemy import PyIntEnum @@ -30,21 +29,22 @@ from indico.modules.users.models.affiliations import UserAffiliation from indico.modules.users.models.emails import UserEmail from indico.modules.users.models.favorites import favorite_category_table, favorite_user_table +from indico.util.enum import RichIntEnum from indico.util.i18n import _ from indico.util.locators import locator_property -from indico.util.string import format_full_name, format_repr, return_ascii -from indico.util.struct.enum import RichIntEnum +from indico.util.string import format_full_name, format_repr from indico.web.flask.util import url_for class UserTitle(RichIntEnum): - __titles__ = ('', _('Mr'), _('Ms'), _('Mrs'), _('Dr'), _('Prof.')) + __titles__ = ('', _('Mr'), _('Ms'), _('Mrs'), _('Dr'), _('Prof.'), _('Mx')) none = 0 mr = 1 ms = 2 mrs = 3 dr = 4 prof = 5 + mx = 6 class NameFormat(RichIntEnum): @@ -67,7 +67,7 @@ class ProfilePictureSource(int, Enum): custom = 3 -class PersonMixin(object): +class PersonMixin: """Add convenience properties and methods to person classes. Assumes the following attributes exist: @@ -77,7 +77,7 @@ class PersonMixin(object): """ def _get_title(self): - """Return title text""" + """Return title text.""" if self._title is None: return get_default_values(type(self)).get('_title', UserTitle.none).title return self._title.title @@ -157,19 +157,12 @@ def format_display_full_name(user, obj): elif name_format in (NameFormat.f_last, NameFormat.f_last_upper): return obj.get_full_name(last_name_first=False, last_name_upper=upper, abbrev_first_name=True) else: - raise ValueError('Invalid name format: {}'.format(name_format)) + raise ValueError(f'Invalid name format: {name_format}') class User(PersonMixin, db.Model): - """Indico users""" - - # Useful when dealing with both users and groups in the same code - is_group = False - is_single_person = True - is_event_role = False - is_category_role = False - is_registration_form = False - is_network = False + """Indico users.""" + principal_order = 0 principal_type = PrincipalType.user @@ -261,7 +254,7 @@ class User(PersonMixin, db.Model): signing_secret = db.Column( UUID, nullable=False, - default=lambda: unicode(uuid4()) + default=lambda: str(uuid4()) ) #: the user profile picture picture = db.deferred(db.Column( @@ -294,14 +287,16 @@ class User(PersonMixin, db.Model): lazy=False, uselist=False, cascade='all, delete-orphan', - primaryjoin='(User.id == UserEmail.user_id) & UserEmail.is_primary' + primaryjoin='(User.id == UserEmail.user_id) & UserEmail.is_primary', + overlaps='_secondary_emails' ) _secondary_emails = db.relationship( 'UserEmail', lazy=True, cascade='all, delete-orphan', collection_class=set, - primaryjoin='(User.id == UserEmail.user_id) & ~UserEmail.is_primary' + primaryjoin='(User.id == UserEmail.user_id) & ~UserEmail.is_primary', + overlaps='_primary_email' ) _all_emails = db.relationship( 'UserEmail', @@ -362,7 +357,8 @@ class User(PersonMixin, db.Model): uselist=False, cascade='all, delete-orphan', primaryjoin='(User.id == APIKey.user_id) & APIKey.is_active', - back_populates='user' + back_populates='user', + overlaps='old_api_keys' ) #: the previous API keys of the user old_api_keys = db.relationship( @@ -371,7 +367,8 @@ class User(PersonMixin, db.Model): cascade='all, delete-orphan', order_by='APIKey.created_dt.desc()', primaryjoin='(User.id == APIKey.user_id) & ~APIKey.is_active', - back_populates='user' + back_populates='user', + overlaps='api_key' ) #: the identities used by this user identities = db.relationship( @@ -426,7 +423,7 @@ class User(PersonMixin, db.Model): # - modified_abstract_comments (AbstractComment.modified_by) # - modified_abstracts (Abstract.modified_by) # - modified_review_comments (PaperReviewComment.modified_by) - # - oauth_tokens (OAuthToken.user) + # - oauth_app_links (OAuthApplicationUserLink.user) # - owned_rooms (Room.owner) # - paper_competences (PaperCompetence.user) # - paper_reviews (PaperReview.user) @@ -447,62 +444,53 @@ def get_system_user(): @property def as_principal(self): - """The serializable principal identifier of this user""" + """The serializable principal identifier of this user.""" return 'User', self.id @property def identifier(self): - return 'User:{}'.format(self.id) - - @property - def as_avatar(self): - # TODO: remove this after DB is free of Avatars - from indico.modules.users.legacy import AvatarUserWrapper - avatar = AvatarUserWrapper(self.id) - - # avoid garbage collection - avatar.user - return avatar - - as_legacy = as_avatar + return f'User:{self.id}' @property def avatar_bg_color(self): from indico.modules.users.util import get_color_for_username return get_color_for_username(self.full_name) - @property - def avatar_css(self): - return 'background-color: {};'.format(self.avatar_bg_color) - @property def external_identities(self): - """The external identities of the user""" + """The external identities of the user.""" return {x for x in self.identities if x.provider != 'indico'} @property def local_identities(self): - """The local identities of the user""" + """The local identities of the user.""" return {x for x in self.identities if x.provider == 'indico'} @property def local_identity(self): - """The main (most recently used) local identity""" + """The main (most recently used) local identity.""" identities = sorted(self.local_identities, key=attrgetter('safe_last_login_dt'), reverse=True) return identities[0] if identities else None @property def secondary_local_identities(self): - """The local identities of the user except the main one""" + """The local identities of the user except the main one.""" return self.local_identities - {self.local_identity} + @property + def last_login_dt(self): + """The datetime when the user last logged in.""" + if not self.identities: + return None + return max(self.identities, key=attrgetter('safe_last_login_dt')).last_login_dt + @locator_property def locator(self): return {'user_id': self.id} @cached_property def settings(self): - """Returns the user settings proxy for this user""" + """Return the user settings proxy for this user.""" from indico.modules.users import user_settings return user_settings.bind(self) @@ -545,7 +533,9 @@ def has_picture(self): return self.picture_metadata is not None @property - def picture_url(self): + def avatar_url(self): + if self.is_system: + return url_for('assets.image', filename='robot.svg') slug = self.picture_metadata['hash'] if self.picture_metadata else 'default' return url_for('users.user_profile_picture_display', self, slug=slug) @@ -553,12 +543,11 @@ def __contains__(self, user): """Convenience method for `user in user_or_group`.""" return self == user - @return_ascii def __repr__(self): return format_repr(self, 'id', 'email', is_deleted=False, is_pending=False, _text=self.full_name) def can_be_modified(self, user): - """If this user can be modified by the given user""" + """If this user can be modified by the given user.""" return self == user or user.is_admin def iter_identifiers(self, check_providers=False, providers=None): @@ -587,36 +576,40 @@ def iter_identifiers(self, check_providers=False, providers=None): @property def can_get_all_multipass_groups(self): - """Check whether it is possible to get all multipass groups the user is in.""" + """ + Check whether it is possible to get all multipass groups the user is in. + """ return all(multipass.identity_providers[x.provider].supports_get_identity_groups for x in self.identities if x.provider != 'indico' and x.provider in multipass.identity_providers) def iter_all_multipass_groups(self): - """Iterate over all multipass groups the user is in""" + """Iterate over all multipass groups the user is in.""" return itertools.chain.from_iterable(multipass.identity_providers[x.provider].get_identity_groups(x.identifier) for x in self.identities if x.provider != 'indico' and x.provider in multipass.identity_providers) def get_full_name(self, *args, **kwargs): kwargs['_show_empty_names'] = True - return super(User, self).get_full_name(*args, **kwargs) + return super().get_full_name(*args, **kwargs) def make_email_primary(self, email): - """Promotes a secondary email address to the primary email address + """Promote a secondary email address to the primary email address. :param email: an email address that is currently a secondary email """ secondary = next((x for x in self._secondary_emails if x.email == email), None) if secondary is None: raise ValueError('email is not a secondary email address') + old = self.email self._primary_email.is_primary = False db.session.flush() secondary.is_primary = True db.session.flush() + signals.users.primary_email_changed.send(self, old=old, new=email) def reset_signing_secret(self): - self.signing_secret = unicode(uuid4()) + self.signing_secret = str(uuid4()) def synchronize_data(self, refresh=False): """Synchronize the fields of the user from the sync identity. diff --git a/indico/modules/users/models/users_test.py b/indico/modules/users/models/users_test.py index b92efd77163..b33808f74b8 100644 --- a/indico/modules/users/models/users_test.py +++ b/indico/modules/users/models/users_test.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -11,6 +11,7 @@ from speaklater import is_lazy_string from sqlalchemy.exc import IntegrityError +from indico.core import signals from indico.modules.users import User from indico.modules.users.models.users import UserTitle @@ -52,7 +53,7 @@ def test_get_full_name(last_name_first, last_name_upper, abbrev_first_name, expe user.title = UserTitle.mr titled_name = user.get_full_name(last_name_first=last_name_first, last_name_upper=last_name_upper, abbrev_first_name=abbrev_first_name, show_title=True) - assert titled_name == 'Mr {}'.format(expected) + assert titled_name == f'Mr {expected}' @pytest.mark.parametrize(('first_name', 'last_name'), ( @@ -85,17 +86,29 @@ def test_emails(db): def test_make_email_primary(db): - user = User(first_name='Guinea', last_name='Pig', email='guinea@pig.com') - db.session.add(user) - db.session.flush() - with pytest.raises(ValueError): + signal_called = False + + def _signal_fn(sender, old, new): + nonlocal signal_called + signal_called = True + assert sender is user + assert old == 'guinea@pig.com' + assert new == 'tasty@pig.com' + + with signals.users.primary_email_changed.connected_to(_signal_fn): + user = User(first_name='Guinea', last_name='Pig', email='guinea@pig.com') + db.session.add(user) + db.session.flush() + with pytest.raises(ValueError): + user.make_email_primary('tasty@pig.com') + user.secondary_emails = {'tasty@pig.com', 'little@pig.com'} + db.session.flush() + assert not signal_called user.make_email_primary('tasty@pig.com') - user.secondary_emails = {'tasty@pig.com', 'little@pig.com'} - db.session.flush() - user.make_email_primary('tasty@pig.com') - db.session.expire(user) - assert user.email == 'tasty@pig.com' - assert user.secondary_emails == {'guinea@pig.com', 'little@pig.com'} + assert signal_called + db.session.expire(user) + assert user.email == 'tasty@pig.com' + assert user.secondary_emails == {'guinea@pig.com', 'little@pig.com'} def test_deletion(db): @@ -132,7 +145,7 @@ def test_title(db): user.title = UserTitle.prof assert user.title == UserTitle.prof.title assert is_lazy_string(user.title) - assert User.find_one(title=UserTitle.prof) == user + assert User.query.filter_by(title=UserTitle.prof).one() == user @pytest.mark.parametrize(('first_name', 'last_name'), ( diff --git a/indico/modules/users/module.json b/indico/modules/users/module.json index 08e3219e7fe..3bb7a8e5635 100644 --- a/indico/modules/users/module.json +++ b/indico/modules/users/module.json @@ -2,6 +2,7 @@ "name": "users", "partials": { "dashboard": "./dashboard.jsx", - "profile_picture": "./ProfilePicture.jsx" + "profile_picture": "./ProfilePicture.jsx", + "favorites": "./Favorites.jsx" } } diff --git a/indico/modules/users/operations.py b/indico/modules/users/operations.py index 342b90a93ec..3bd1f260e09 100644 --- a/indico/modules/users/operations.py +++ b/indico/modules/users/operations.py @@ -1,12 +1,10 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from indico.core import signals from indico.core.config import config from indico.core.db import db diff --git a/indico/modules/users/schemas.py b/indico/modules/users/schemas.py index 62fda35d2e2..6670d666626 100644 --- a/indico/modules/users/schemas.py +++ b/indico/modules/users/schemas.py @@ -1,25 +1,27 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import unicode_literals - from marshmallow.fields import Function from indico.core.marshmallow import mm +from indico.modules.categories import Category from indico.modules.users import User -class UserSchema(mm.ModelSchema): +class UserSchema(mm.SQLAlchemyAutoSchema): identifier = Function(lambda user: user.identifier) class Meta: model = User - fields = ('id', 'identifier', 'first_name', 'last_name', 'email', 'affiliation', 'avatar_bg_color', 'full_name', - 'phone') + fields = ('id', 'identifier', 'first_name', 'last_name', 'email', 'affiliation', 'full_name', + 'phone', 'avatar_url') -user_schema = UserSchema() +class BasicCategorySchema(mm.SQLAlchemyAutoSchema): + class Meta: + model = Category + fields = ('id', 'title', 'url', 'chain_titles') diff --git a/indico/modules/users/tasks.py b/indico/modules/users/tasks.py index b849257464f..75977bdcc73 100644 --- a/indico/modules/users/tasks.py +++ b/indico/modules/users/tasks.py @@ -1,5 +1,5 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the @@ -12,8 +12,8 @@ from indico.modules.users import logger from indico.modules.users.models.users import ProfilePictureSource, User from indico.modules.users.util import get_gravatar_for_user, set_user_avatar +from indico.util.iterables import committing_iterator from indico.util.string import crc32 -from indico.util.struct.iterables import committing_iterator @celery.periodic_task(name='update_gravatars', run_every=crontab(minute='0', hour='0')) diff --git a/indico/modules/users/templates/_category.html b/indico/modules/users/templates/_category.html index 8d823fcac75..e945347b907 100644 --- a/indico/modules/users/templates/_category.html +++ b/indico/modules/users/templates/_category.html @@ -61,14 +61,3 @@ {% endcall %} {% endmacro %} - -{% macro favorite_category(category, truncated_path) %} - {% call _category_item(category, truncated_path) %} - - - - {% endcall %} -{% endmacro %} diff --git a/indico/modules/users/templates/_favorites.html b/indico/modules/users/templates/_favorites.html deleted file mode 100644 index 348f9dbcd1c..00000000000 --- a/indico/modules/users/templates/_favorites.html +++ /dev/null @@ -1,12 +0,0 @@ -{% macro favorite_users_list(user) %} - {% for fav in user.favorite_users|sort(attribute='full_name') %} -
      • - {{ fav.full_name }} - - - -
        • - {% endfor %} -{% endmacro %} diff --git a/indico/modules/users/templates/dashboard.html b/indico/modules/users/templates/dashboard.html index 6beca02dccf..13389e903ae 100644 --- a/indico/modules/users/templates/dashboard.html +++ b/indico/modules/users/templates/dashboard.html @@ -28,7 +28,7 @@ {%- endblock %} {% block banner_actions -%} - + {% endblock %} {% block content %} @@ -37,7 +37,7 @@
        - +

        {{ user.full_name }}

        @@ -49,6 +49,17 @@

        {{ user.full_name }}

        {% endif %} {{ labels(user) }} +
        +
        + {% trans %}Last login{% endtrans %} +
        + {% if user.last_login_dt %} + {{ user.last_login_dt | format_datetime }} + {% else %} + {% trans %}Never{% endtrans %} + {% endif %} +
        +
        {% if user.affiliation %}
        @@ -127,7 +138,7 @@

        {% trans %}Your categories{% endtrans %}

        • {% else %} - {% for category in categories.itervalues() %} + {% for category in categories.values() %} {{ user_category(category.categ, category.path, category.managed) }} {% endfor %} {% endif %} diff --git a/indico/modules/users/templates/emails/verify_email.txt b/indico/modules/users/templates/emails/verify_email.txt index 1b07deada09..37de478c9ef 100644 --- a/indico/modules/users/templates/emails/verify_email.txt +++ b/indico/modules/users/templates/emails/verify_email.txt @@ -9,12 +9,10 @@ {%- endblock %} {% block body -%} - {%- filter dedent -%} - {%- trans -%} - You recently added the e-mail address {{ email }} to your Indico account. - Please click the following link to confirm this action. - {%- endtrans %} +{%- trans notrimmed -%} +You recently added the e-mail address {{ email }} to your Indico account. +Please click the following link to confirm this action. +{%- endtrans %} - {{ url_for('.user_emails_verify', token=token, _external=true) }} - {%- endfilter -%} +{{ url_for('.user_emails_verify', token=token, _external=true) }} {%- endblock %} diff --git a/indico/modules/users/templates/favorites.html b/indico/modules/users/templates/favorites.html index 381b6c358fe..638e1ec5f50 100644 --- a/indico/modules/users/templates/favorites.html +++ b/indico/modules/users/templates/favorites.html @@ -1,97 +1,14 @@ {% extends 'users/base.html' %} -{% from 'users/_favorites.html' import favorite_users_list %} -{% from 'users/_category.html' import favorite_category %} {% block user_content %}
        -
        -
        -
        -
        - {%- trans %}Favourite Users{% endtrans -%} -
        -
        -
        - {% if user.favorite_users %} -
          - {{ favorite_users_list(user) }} -
        - {% else %} - - {% trans %}You have not marked any user as favourite.{% endtrans %} - - {% endif %} -
        -
        - -
        -
        -
        -
        -
        - {%- trans %}Favourite Categories{% endtrans -%} -
        -
        -
        - {% if favorite_categories %} -
          - {% for category, truncated_path in favorite_categories %} - {{ favorite_category(category, truncated_path) }} - {% endfor %} -
        - {% else %} - - {% trans %}You have not marked any category as favourite.{% endtrans %} - - {% endif %} -
        -
        -
        +
        -{% endblock %} diff --git a/indico/web/templates/_protection_messages.html b/indico/web/templates/_protection_messages.html index 10c5e2250b6..cdc4e7e477f 100644 --- a/indico/web/templates/_protection_messages.html +++ b/indico/web/templates/_protection_messages.html @@ -61,6 +61,7 @@

    +
    {%- endmacro %} {%- macro render_non_inheriting_children_message(protected_object, non_inheriting_objects) -%} diff --git a/indico/web/templates/_session_bar.html b/indico/web/templates/_session_bar.html index 2ebd26dcd85..f109564e6cb 100644 --- a/indico/web/templates/_session_bar.html +++ b/indico/web/templates/_session_bar.html @@ -2,7 +2,7 @@ {% if not obj.is_protected %} {% set mode = 'public' %} {% else %} - {% set networks = obj.get_access_list()|selectattr('is_network')|map(attribute='name')|sort %} + {% set networks = obj.get_access_list()|selectattr('principal_type.name', 'equalto', 'network')|map(attribute='name')|sort %} {% set mode = 'network' if networks else 'restricted' %} {% endif %} @@ -134,9 +134,7 @@
    {% if session.user.is_admin %}
    - +
    {% endif %} {% if 'login_as_orig_user' in session %} diff --git a/indico/web/templates/_sortable_list.html b/indico/web/templates/_sortable_list.html index 339c6096e5b..42b2dcacbaa 100644 --- a/indico/web/templates/_sortable_list.html +++ b/indico/web/templates/_sortable_list.html @@ -14,7 +14,7 @@
    - {% if caller %} + {% if caller is defined %} {{ caller(item) }} {% endif %}
    @@ -24,7 +24,7 @@ {% macro sortable_list(items, classes="", draggable=true, title=none, id=none, invisible_handle=false) -%} {# Caller is called with an item and must render the action buttons. #} - {% set _caller = caller %} + {% set _caller = caller|default(none) %}
    @@ -44,7 +44,7 @@ {% macro sortable_lists(enabled_title, enabled_items, disabled_title, disabled_items, classes="", draggable=true, id=none, invisible_handle=false) -%} {# Caller is called with an item and must render the action buttons. #} - {% set _caller = caller %} + {% set _caller = caller|default(none) %}
    diff --git a/indico/web/templates/_statistics.html b/indico/web/templates/_statistics.html index 0e77829a291..9eaba0a82d5 100644 --- a/indico/web/templates/_statistics.html +++ b/indico/web/templates/_statistics.html @@ -1,6 +1,6 @@ {% macro stats_box(title='', subtitle='', label='', label_tooltip='', classes='') %}
    + {%- if classes %} {{ classes }}{% endif %}"> {%- if title or label %}
    {% if title -%} diff --git a/indico/web/templates/bad_url_error.html b/indico/web/templates/bad_url_error.html new file mode 100644 index 00000000000..13479ffd743 --- /dev/null +++ b/indico/web/templates/bad_url_error.html @@ -0,0 +1,9 @@ +{% set error_message = _('Invalid URL') %} + +{% set error_description -%} + {%- trans url=indico_config.BASE_URL -%} + This Indico instance can only be accessed via {{ url }} + {%- endtrans -%} +{%- endset %} + +{% include 'standalone_error.html' %} diff --git a/indico/web/templates/error.html b/indico/web/templates/error.html index ba73f0ee0fb..a6af95f18b8 100644 --- a/indico/web/templates/error.html +++ b/indico/web/templates/error.html @@ -1,7 +1,7 @@

    {{ error_message }}

    {{ error_description }}

    - {% if g.saved_error_uuid and not standalone %} + {% if g.saved_error_uuid and not standalone|default(false) %} diff --git a/indico/web/templates/footer.html b/indico/web/templates/footer.html index 75645cbb45c..38a01c71b74 100644 --- a/indico/web/templates/footer.html +++ b/indico/web/templates/footer.html @@ -1,4 +1,4 @@ -
    +
    {% block footer -%} {%- set indico -%} Indico {{ indico_version }} @@ -6,7 +6,7 @@
    {% block footer_logo %} - {% set filename = 'indico_small_white.png' if dark else 'indico_small.png' %} + {% set filename = 'indico_small_white.png' if dark|default(false) else 'indico_small.png' %} {% endblock %}
    diff --git a/indico/web/templates/forms/_form.html b/indico/web/templates/forms/_form.html index 99dc5a9ff47..823d8ad961e 100644 --- a/indico/web/templates/forms/_form.html +++ b/indico/web/templates/forms/_form.html @@ -14,7 +14,7 @@ class="{{ 'i-form' if i_form }} {{ 'disable-fields-if-locked' if disable_if_locked }} {{ orientation }} {{ classes }}" {% if id %}id="{{ id }}"{% endif %} {% if multipart %}enctype="multipart/form-data"{% endif %} - {% if caller %}{{ caller() }}{% endif %} + {% if caller is defined %}{{ caller() }}{% endif %} {% if no_ajax %}data-no-ajax{% endif %} {{ extra_attrs|html_params }}> {% if form.meta.csrf -%} @@ -48,7 +48,7 @@ {% if field.widget.input_type == 'checkbox' %}
    {{ field.label() }}
    {% endif %} - {%- if caller -%} + {%- if caller is defined -%} {{ caller() }} {%- endif -%} {% if not hide_description and field.description %} @@ -81,7 +81,7 @@ {% endif %} {% set widget_attrs = dict(widget_attrs, placeholder=(field.label.text if placeholder else widget_attrs.get('placeholder', ''))) %} {% set classes = 'form-field %s' | format(classes) %} - {%- if caller -%} + {%- if caller is defined -%} {% set caller_ = caller %} {% call form_field(field, classes, widget_attrs, disabled=disabled, hide_description=hide_description) -%} {{ caller_() }} @@ -102,7 +102,7 @@ {# `form` and `i_form` args are there in case we need them in the future #} {% macro form_footer(form, i_form=true, skip_label=false, align_right=false, classes='') -%} - {% if caller %} + {% if caller is defined %}
    {% if not skip_label -%}
    @@ -160,7 +160,7 @@ {% if message %} {{ message_box('warning', message=message, icon=true) }} {% endif %} - {% if caller %} + {% if caller is defined %} {{ caller() }} {% else %} {{ form_rows(form, fields=fields, disable=disabled_fields, skip_labels=skip_labels) }} diff --git a/indico/web/templates/forms/_person_link_widget_base.html b/indico/web/templates/forms/_person_link_widget_base.html index fc86cf708b9..a202bcb94e1 100644 --- a/indico/web/templates/forms/_person_link_widget_base.html +++ b/indico/web/templates/forms/_person_link_widget_base.html @@ -35,10 +35,8 @@ {% trans %}Add myself{% endtrans %} {% endif %} - - {% trans %}Search{% endtrans %} - + + {#--#} {% trans %}Enter manually{% endtrans %} diff --git a/indico/web/templates/forms/category_picker_widget.html b/indico/web/templates/forms/category_picker_widget.html index 047087a8cae..9d25f8b520e 100644 --- a/indico/web/templates/forms/category_picker_widget.html +++ b/indico/web/templates/forms/category_picker_widget.html @@ -19,10 +19,6 @@ fieldId: {{ field.id | tojson }}, navigatorCategoryId: {{ field.navigator_category_id | tojson }}, requireEventCreationRights: {{ field.require_event_creation_rights | tojson }}, - allow: { - events: {{ field.allow_events | tojson }}, - subcats: {{ field.allow_subcats | tojson }} - } }); {% endblock %} diff --git a/indico/web/templates/forms/checkbox_group_widget.html b/indico/web/templates/forms/checkbox_group_widget.html index 23f6fcc2295..aa71bd73861 100644 --- a/indico/web/templates/forms/checkbox_group_widget.html +++ b/indico/web/templates/forms/checkbox_group_widget.html @@ -5,7 +5,7 @@ {% set input_args = dict(input_args) %} {% set field_disabled = input_args.pop('disabled', false) %} {% for subfield in field %} - {% set disabled = field_disabled or subfield.data in field.disabled_choices %} + {% set disabled = field_disabled or subfield.data in field.disabled_choices|default([]) %}
    {{ subfield(disabled=disabled, **input_args) }} {{ subfield.label }} diff --git a/indico/web/templates/forms/datetime_widget.html b/indico/web/templates/forms/datetime_widget.html index 0bc3885d46b..23a52da9d7c 100644 --- a/indico/web/templates/forms/datetime_widget.html +++ b/indico/web/templates/forms/datetime_widget.html @@ -8,7 +8,8 @@ {% if field.data and not field.date_missing %}value="{{ field.data | format_date('code', timezone=field.timezone) }}"{% endif %} {{ input_args | html_params }}>{#--#} + {% if field.data and not field.time_missing %}value="{{ field.data | format_time('code', timezone=field.timezone) }}"{% endif %} + {% if input_args.disabled %}disabled{% endif %}>
    {% endblock %} diff --git a/indico/web/templates/forms/permissions_widget.html b/indico/web/templates/forms/permissions_widget.html index d6cc969099d..d013c2399d7 100644 --- a/indico/web/templates/forms/permissions_widget.html +++ b/indico/web/templates/forms/permissions_widget.html @@ -10,7 +10,7 @@

    Permissions

    {%- trans %}Add{% endtrans -%} - {% trans %}User / Group{% endtrans %} + {% if field.object_type == 'event' %} {% trans %}Event Role{% endtrans %}
      diff --git a/indico/web/templates/forms/prefixed_text_widget.html b/indico/web/templates/forms/prefixed_text_widget.html new file mode 100644 index 00000000000..87304ed226a --- /dev/null +++ b/indico/web/templates/forms/prefixed_text_widget.html @@ -0,0 +1,9 @@ +{% extends 'forms/base_widget.html' %} + +{% block html %} +
      + {{ prefix }} + +
      +{% endblock %} diff --git a/indico/web/templates/forms/principal_list_widget.html b/indico/web/templates/forms/principal_list_widget.html index 47c5d071225..d8ba2162266 100644 --- a/indico/web/templates/forms/principal_list_widget.html +++ b/indico/web/templates/forms/principal_list_widget.html @@ -2,17 +2,8 @@ {% block html %} - {% if acl %} - - {% endif %} {% set value_json = field._value() | tojson %} -
      - {% if acl %} -
      - {% endif %} +
      diff --git a/indico/web/templates/forms/principal_widget.html b/indico/web/templates/forms/principal_widget.html index 3365f8eb95d..b53e0a34cd3 100644 --- a/indico/web/templates/forms/principal_widget.html +++ b/indico/web/templates/forms/principal_widget.html @@ -13,7 +13,8 @@ setupPrincipalWidget({ fieldId: {{ field.id | tojson }}, required: {{ input_args.required | default(false) | tojson }}, - withExternalUsers: {{ field.allow_external_users | tojson }} + withExternalUsers: {{ field.allow_external_users | tojson }}, + disabled: {{ (field.render_kw.disabled if field.render_kw else false) | tojson }} }); {% endblock %} diff --git a/indico/web/templates/forms/radio_buttons_widget.html b/indico/web/templates/forms/radio_buttons_widget.html index ca8edb85941..b502441f698 100644 --- a/indico/web/templates/forms/radio_buttons_widget.html +++ b/indico/web/templates/forms/radio_buttons_widget.html @@ -4,9 +4,7 @@ {% block html %}
      - {% if field.enum and field.enum.__css_classes__ %} - {% set options = field.enum | list %} - {% endif %} + {% set options = field.enum|list if field.enum and field.enum.__css_classes__ else none %} {% for subfield in field %}
      {{ subfield(**input_args) }} diff --git a/indico/web/templates/indico_base.html b/indico/web/templates/indico_base.html index 832f3cc3518..a3d2594d139 100644 --- a/indico/web/templates/indico_base.html +++ b/indico/web/templates/indico_base.html @@ -54,7 +54,7 @@ - {{ head_content|safe }} + {{ head_content|default('')|safe }} {% for url in css_files -%} diff --git a/indico/web/templates/meta.html b/indico/web/templates/meta.html index 808c84413f5..03d00f87af1 100644 --- a/indico/web/templates/meta.html +++ b/indico/web/templates/meta.html @@ -1,5 +1,5 @@ {%- if page_metadata.og -%} - {% for key, value in page_metadata.og.viewitems() -%} + {% for key, value in page_metadata.og.items() -%} {% endfor %} {%- endif %} diff --git a/indico/web/templates/overview/base.html b/indico/web/templates/overview/base.html index bfb665f7ae4..24777f94c2e 100644 --- a/indico/web/templates/overview/base.html +++ b/indico/web/templates/overview/base.html @@ -15,7 +15,7 @@ {% block content %}
      - {{ form_header(form, method='get', action=action) }} + {{ form_header(form, method='get', action=action|default('')) }} {{ form_rows(form) }} {% call form_footer(form) %} @@ -35,7 +35,7 @@ {% else %} {% block result_list %}
    {{ data.get(item.id).friendly_data }}{{ data[item.id].friendly_data if item.id in data }}
    - {% for key, items in results.iteritems() %} + {% for key, items in results.items() %}
    {{ result_group_title(key) if result_group_title is defined else key }} diff --git a/indico/web/templates/placeholder_info.html b/indico/web/templates/placeholder_info.html index 8a90eb1ec7f..4be19ee9550 100644 --- a/indico/web/templates/placeholder_info.html +++ b/indico/web/templates/placeholder_info.html @@ -8,7 +8,7 @@ {% for placeholder in placeholders if placeholder.advanced == advanced and placeholder is subclassof ParametrizedPlaceholder -%} {% for param, description in placeholder.iter_param_info(**placeholder_kwargs) %} {{ br() }} - { {{- placeholder.name -}}{%- if param is not none -%}:{{- param -}}{%- endif -%} }: + { {{- placeholder.name -}}{%- if param is not none -%}:{{- param -}}{%- endif -%} } - {{ description }} {% if placeholder.required %}(required){% endif %} {% endfor %} {% endfor %} diff --git a/indico/web/templates/standalone_error.html b/indico/web/templates/standalone_error.html index a07198ece96..cb0c74ecf34 100644 --- a/indico/web/templates/standalone_error.html +++ b/indico/web/templates/standalone_error.html @@ -57,7 +57,7 @@
    - +
    {% with standalone=true %} diff --git a/indico/web/util.py b/indico/web/util.py index 8c6810a836c..dcf1932bd77 100644 --- a/indico/web/util.py +++ b/indico/web/util.py @@ -1,26 +1,29 @@ # This file is part of Indico. -# Copyright (C) 2002 - 2020 CERN +# Copyright (C) 2002 - 2021 CERN # # Indico is free software; you can redistribute it and/or # modify it under the terms of the MIT License; see the # LICENSE file for more details. -from __future__ import absolute_import, unicode_literals - +import hashlib +import sys from datetime import datetime -from flask import g, has_request_context, jsonify, render_template, request, session +import sentry_sdk +from authlib.oauth2 import OAuth2Error +from flask import flash, g, has_request_context, jsonify, render_template, request, session from itsdangerous import Signer from markupsafe import Markup -from werkzeug.exceptions import ImATeapot +from werkzeug.exceptions import BadRequest, Forbidden, ImATeapot from werkzeug.urls import url_decode, url_encode, url_parse, url_unparse +from indico.util.caching import memoize_request from indico.util.i18n import _ from indico.web.flask.templating import get_template_module def inject_js(js): - """Injects JavaScript into the current page. + """Inject JavaScript into the current page. :param js: Code wrapped in a ``