Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Template improvements #3760

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
6 changes: 6 additions & 0 deletions CHANGES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ New in master
Features
--------

* Trace template usage when an environment variable ``NIKOLA_TEMPLATES_TRACE``
is set to any non-empty value.
* Give user control over the raw underlying template engine
(either `mako.lookup.TemplateLookup` or `jinja2.Environment`)
via an optional `conf.py` method `TEMPLATE_ENGINE_FACTORY`.

Bugfixes
--------

Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,4 @@ Here are some guidelines about how you can contribute to Nikola:

.. [1] Very inspired by `fabric’s <https://github.com/fabric/fabric/blob/master/CONTRIBUTING.rst>`_ — thanks!

.. [2] For example, logging or always making sure directories are created using ``utils.makedirs()``.
.. [2] For example, logging, or always making sure directories are created using ``utils.makedirs()``.
55 changes: 47 additions & 8 deletions docs/theming.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
.. author: The Nikola Team

:Version: 8.3.1
:Author: Roberto Alsina <[email protected]>
:Author: Roberto Alsina <[email protected]> and others

.. class:: alert alert-primary float-md-right

Expand Down Expand Up @@ -130,8 +130,17 @@ The following keys are currently supported:

The parent is so you don’t have to create a full theme each time: just
create an empty theme, set the parent, and add the bits you want modified.
You **must** define a parent, otherwise many features won’t work due to
missing templates, messages, and assets.

While it is possible to create a theme without a parent, it is
**strongly discouraged** and not officially supported, in the sense:
We won't help with issues that are caused by a theme being parentless,
and we won't guarantee that it will always work with new Nikola versions.
The `base` and `base-jinja` themes provide assets, messages, and generic templates
that Nikola expects to be able to use in all sites. That said, if you are making
something very custom, Nikola will not prevent the creation of a theme
without `base`, but you will need to manually determine which templates and
messages are required in your theme. (Initially setting the ``NIKOLA_TEMPLATES_TRACE``
environment variable might be of some help, see below.)

The following settings are recommended:

Expand Down Expand Up @@ -184,8 +193,16 @@ so ``post.tmpl`` only define the content, and the layout is inherited from ``bas

Another concept is theme inheritance. You do not need to duplicate all the
default templates in your theme — you can just override the ones you want
changed, and the rest will come from the parent theme. (Every theme needs a
parent.)
changed, and the rest will come from the parent theme. If your theme does not
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, unsupported and should not be documented.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about the compromise "we document that this is unsupported", which is what I'm trying to do here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m still not a fan of documenting it at all, but I guess it can stay as-is.

define a parent, it needs to be complete. It is generally a lot harder to
come up with a complete theme, compared to only changing a few files and using
the rest from a suitable parent theme.

.. Tip::

If you set the environment variable ``NIKOLA_TEMPLATES_TRACE`` to any non-empty value
(``true`` is recommended), Nikola will log template usage, both on output and also
into a file ``templates_log.txt``.
Comment on lines +203 to +205
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary detail.

Suggested change
If you set the environment variable ``NIKOLA_TEMPLATES_TRACE`` to any non-empty value
(``true`` is recommended), Nikola will log template usage, both on output and also
into a file ``templates_log.txt``.
If you set the environment variable ``NIKOLA_TEMPLATES_TRACE`` to ``true``,
Nikola will log template usage, both on output and also into a file ``templates_log.txt``.


Apart from the `built-in templates`_ listed below, you can add other templates for specific
pages, which the user can then use in his ``POSTS`` or ``PAGES`` option in
Expand All @@ -194,11 +211,11 @@ page via the ``template`` metadata, and custom templates can be added in the
``templates/`` folder of your site.

If you want to modify (override) a built-in template, use ``nikola theme -c
<name>.tmpl``. This command will copy the specified template file to the
``templates/`` directory of your currently used theme.
<name>.tmpl``. This command will copy the specified template file from the
parent theme to the ``templates/`` directory of your currently used theme.

Keep in mind that your theme is *yours*, so you can require whatever data you
want (eg. you may depend on specific custom ``GLOBAL_CONTEXT`` variables, or
want (e.g., you may depend on specific custom ``GLOBAL_CONTEXT`` variables, or
post meta attributes). You don’t need to keep the same theme structure as the
default themes do (although many of those names are hardcoded). Inheriting from
at least ``base`` (or ``base-jinja``) is heavily recommended, but not strictly
Expand Down Expand Up @@ -475,6 +492,28 @@ at https://www.transifex.com/projects/p/nikola/
If you want to create a theme that has new strings, and you want those strings to be translatable,
then your theme will need a custom ``messages`` folder.

Configuration of the raw template engine
----------------------------------------

For usage not covered by the above, you can define a method
`TEMPLATE_ENGINE_FACTORY` in `conf.py` that constructs the raw
underlying templating engine. That `raw_engine` that your method
needs to return is either a `jinja2.Environment` or a
`mako.loopkup.TemplateLookup` object. Your factory method is
called with the same arguments as is the pertinent `__init__`.

E.g., to configure `jinja2` to bark and error out on missing values,
instead of silently continuing with empty content, you might do this:

.. code:: python

# Somewhere in conf.py:
def TEMPLATE_ENGINE_FACTORY(**args) -> jinja2.Environment:
augmented_args = dict(args)
augmented_args['undefined'] = jinja2.DebugUndefined
return jinja2.Environment(**augmented_args)


`LESS <http://lesscss.org/>`__ and `Sass <https://sass-lang.com/>`__
--------------------------------------------------------------------

Expand Down
11 changes: 11 additions & 0 deletions nikola/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,19 @@
import os
import sys

# The current Nikola version:
__version__ = '8.3.1'

# A flag whether logging should emit debug information:
DEBUG = bool(os.getenv('NIKOLA_DEBUG'))

# A flag whether special templates trace logging should be generated:
aknrdureegaesr marked this conversation as resolved.
Show resolved Hide resolved
TEMPLATES_TRACE = bool(os.getenv('NIKOLA_TEMPLATES_TRACE'))

# When this flag is set, fewer exceptions are handled internally;
# instead they are left unhandled for the run time system to deal with them,
# which typically leads to the stack traces being exposed.
Comment on lines +41 to +43
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# When this flag is set, fewer exceptions are handled internally;
# instead they are left unhandled for the run time system to deal with them,
# which typically leads to the stack traces being exposed.
# A flag to show tracebacks of unhandled exceptions.
# An alternative to the DEBUG flag with less noise.

(NIKOLA_DEBUG used to be very spammy with yapsy, it’s still not great to have it always on; export NIKOLA_SHOW_TRACEBACKS=1 is in my .zshrc though.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wording "of unhandled exceptions" suggests whether the exceptions is handled or not is independent of this setting. As far as I remember, this is not the case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically, all exceptions are handled. This setting affects exceptions that bubble up to the global handler and therefore crash the process.

# This is a less noisy alternative to the NIKOLA_DEBUG flag.
SHOW_TRACEBACKS = bool(os.getenv('NIKOLA_SHOW_TRACEBACKS'))

if sys.version_info[0] == 2:
Expand Down
14 changes: 14 additions & 0 deletions nikola/conf.py.in
Original file line number Diff line number Diff line change
Expand Up @@ -1319,6 +1319,20 @@ WARN_ABOUT_TAG_METADATA = False
# those.
# TEMPLATE_FILTERS = {}

# If you want to, you can augment or change Nikola's configuration
# of the underlying template engine used
# in any way you please, by defining this function:
# def TEMPLATE_ENGINE_FACTORY(**args):
# pass
# This should return either a jinja2.Environment or a mako.lookup.TemplateLookup
# object that have been configured with the args received plus any additional configuration wanted.
#
# E.g., to configure Jinja2 to bark on non-existing values instead of silently omitting:
# def TEMPLATE_ENGINE_FACTORY(**args) -> jinja2.Environment:
# augmented_args = dict(args)
# augmented_args['undefined'] = jinja2.DebugUndefined
# return jinja2.Environment(**augmented_args)

# Put in global_context things you want available on all your templates.
# It can be anything, data, functions, modules, etc.
GLOBAL_CONTEXT = {}
Expand Down
46 changes: 39 additions & 7 deletions nikola/log.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
import logging
import warnings

from nikola import DEBUG
from nikola import DEBUG, TEMPLATES_TRACE

__all__ = (
"get_logger",
Expand Down Expand Up @@ -86,6 +86,10 @@ class LoggingMode(enum.Enum):
QUIET = 2


_LOGGING_FMT = "[%(asctime)s] %(levelname)s: %(name)s: %(message)s"
_LOGGING_DATEFMT = "%Y-%m-%d %H:%M:%S"


def configure_logging(logging_mode: LoggingMode = LoggingMode.NORMAL) -> None:
"""Configure logging for Nikola.

Expand All @@ -101,12 +105,7 @@ def configure_logging(logging_mode: LoggingMode = LoggingMode.NORMAL) -> None:
return

handler = logging.StreamHandler()
handler.setFormatter(
ColorfulFormatter(
fmt="[%(asctime)s] %(levelname)s: %(name)s: %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
)
handler.setFormatter(ColorfulFormatter(fmt=_LOGGING_FMT, datefmt=_LOGGING_DATEFMT))

handlers = [handler]
if logging_mode == LoggingMode.STRICT:
Expand Down Expand Up @@ -137,6 +136,39 @@ def get_logger(name: str, handlers=None) -> logging.Logger:


LOGGER = get_logger("Nikola")
TEMPLATES_LOGGER = get_logger("nikola.templates")


def init_template_trace_logging(filename: str) -> None:
"""Initialize the tracing of the template system.

This tells a theme designer which templates are being exercised
and for which output files, and, if applicable, input files.

As there is lots of other stuff happening on the normal output stream,
this info is also written to a log file.
"""
TEMPLATES_LOGGER.level = logging.DEBUG
formatter = logging.Formatter(
fmt=_LOGGING_FMT,
datefmt=_LOGGING_DATEFMT,
)
shandler = logging.StreamHandler()
shandler.setFormatter(formatter)
shandler.setLevel(logging.DEBUG)

fhandler = logging.FileHandler(filename, encoding="UTF-8")
fhandler.setFormatter(formatter)
fhandler.setLevel(logging.DEBUG)

TEMPLATES_LOGGER.handlers = [shandler, fhandler]
TEMPLATES_LOGGER.propagate = False

TEMPLATES_LOGGER.info("Template usage being traced to file %s", filename)


if DEBUG or TEMPLATES_TRACE:
init_template_trace_logging("templates_trace.log")


# Push warnings to logging
Expand Down
32 changes: 21 additions & 11 deletions nikola/nikola.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import pathlib
import sys
import typing
from typing import Any, Callable, Dict, Iterable, List, Optional, Set
import mimetypes
from collections import defaultdict
from copy import copy
Expand Down Expand Up @@ -373,7 +374,7 @@ class Nikola(object):
plugin_manager: PluginManager
_template_system: TemplateSystem

def __init__(self, **config):
def __init__(self, **config) -> None:
"""Initialize proper environment for running tasks."""
# Register our own path handlers
self.path_handlers = {
Expand All @@ -395,7 +396,7 @@ def __init__(self, **config):
self.timeline = []
self.pages = []
self._scanned = False
self._template_system = None
self._template_system: Optional[TemplateSystem] = None
self._THEMES = None
self._MESSAGES = None
self.filters = {}
Expand Down Expand Up @@ -996,13 +997,13 @@ def __init__(self, **config):
# WebP files have no official MIME type yet, but we need to recognize them (Issue #3671)
mimetypes.add_type('image/webp', '.webp')

def _filter_duplicate_plugins(self, plugin_list: typing.Iterable[PluginCandidate]):
def _filter_duplicate_plugins(self, plugin_list: Iterable[PluginCandidate]):
"""Find repeated plugins and discard the less local copy."""
def plugin_position_in_places(plugin: PluginInfo):
# plugin here is a tuple:
# (path to the .plugin file, path to plugin module w/o .py, plugin metadata)
place: pathlib.Path
for i, place in enumerate(self._plugin_places):
place: pathlib.Path
try:
# Path.is_relative_to backport
plugin.source_dir.relative_to(place)
Expand All @@ -1025,7 +1026,7 @@ def plugin_position_in_places(plugin: PluginInfo):
result.append(plugins[-1])
return result

def init_plugins(self, commands_only=False, load_all=False):
def init_plugins(self, commands_only=False, load_all=False) -> None:
"""Load plugins as needed."""
extra_plugins_dirs = self.config['EXTRA_PLUGINS_DIRS']
self._loading_commands_only = commands_only
Expand Down Expand Up @@ -1086,9 +1087,9 @@ def init_plugins(self, commands_only=False, load_all=False):
# Search for compiler plugins which we disabled but shouldn't have
self._activate_plugins_of_category("PostScanner")
if not load_all:
file_extensions = set()
file_extensions: Set[str] = set()
post_scanner: PostScanner
for post_scanner in [p.plugin_object for p in self.plugin_manager.get_plugins_of_category('PostScanner')]:
post_scanner: PostScanner
exts = post_scanner.supported_extensions()
if exts is not None:
file_extensions.update(exts)
Expand Down Expand Up @@ -1126,8 +1127,8 @@ def init_plugins(self, commands_only=False, load_all=False):

self._activate_plugins_of_category("Taxonomy")
self.taxonomy_plugins = {}
taxonomy: Taxonomy
for taxonomy in [p.plugin_object for p in self.plugin_manager.get_plugins_of_category('Taxonomy')]:
taxonomy: Taxonomy
if not taxonomy.is_enabled():
continue
if taxonomy.classification_name in self.taxonomy_plugins:
Expand Down Expand Up @@ -1322,7 +1323,7 @@ def _activate_plugin(self, plugin_info: PluginInfo) -> None:
if candidate.exists() and candidate.is_dir():
self.template_system.inject_directory(str(candidate))

def _activate_plugins_of_category(self, category) -> typing.List[PluginInfo]:
def _activate_plugins_of_category(self, category) -> List[PluginInfo]:
"""Activate all the plugins of a given category and return them."""
# this code duplicated in tests/base.py
plugins = []
Expand Down Expand Up @@ -1397,6 +1398,11 @@ def _get_template_system(self):
"plugin\n".format(template_sys_name))
sys.exit(1)
self._template_system = typing.cast(TemplateSystem, pi.plugin_object)

engine_factory: Optional[Callable[..., Any]] = self.config.get("TEMPLATE_ENGINE_FACTORY")
if engine_factory is not None:
self._template_system.user_engine_factory(engine_factory)

lookup_dirs = ['templates'] + [os.path.join(utils.get_theme_path(name), "templates")
for name in self.THEMES]
self._template_system.set_directories(lookup_dirs,
Expand Down Expand Up @@ -1444,7 +1450,7 @@ def get_compiler(self, source_name):

return compiler

def render_template(self, template_name, output_name, context, url_type=None, is_fragment=False):
def render_template(self, template_name: str, output_name: str, context, url_type=None, is_fragment=False):
"""Render a template with the global context.

If ``output_name`` is None, will return a string and all URL
Expand All @@ -1459,7 +1465,11 @@ def render_template(self, template_name, output_name, context, url_type=None, is
If ``is_fragment`` is set to ``True``, a HTML fragment will
be rendered and not a whole HTML document.
"""
local_context = {}
if "post" in context and context["post"] is not None:
utils.TEMPLATES_LOGGER.debug("For %s, template %s builds %s", context["post"].source_path, template_name, output_name)
else:
utils.TEMPLATES_LOGGER.debug("Template %s builds %s", template_name, output_name)
local_context: Dict[str, Any] = {}
local_context["template_name"] = template_name
local_context.update(self.GLOBAL_CONTEXT)
local_context.update(context)
Expand Down
Loading
Loading