New drivers and co

README.rst

@@ -5,22 +5,36 @@
 
 .. From here shared with index.rst of docs folder. #SHARED_CONTENT
 
-Sphinx-Collections
-==================
+.. image:: https://github.com/useblocks/sphinx-collections/raw/master/docs/_static/sphinx_collections_logo.png
+   :align: center
+
+``Sphinx-Collections`` is a Sphinx extension to collect and generate additional files from different sources.
+These files are added to the Sphinx Source Folder, so that Sphinx takes them into account for the overall
+documentation build.
+
+.. image:: https://github.com/useblocks/sphinx-collections/raw/master/docs/_static/sphinx_collections_chart.png
+   :align: center
+
+``Sphinx Collections`` supports multiple collections, where each collection has its own
+source and target folder, specific configuration and use case based driver.
 
-``Sphinx-Collections`` is a Sphinx extension to collect and generate additional files from different sources before
-Sphinx starts the overall build.
+A collection can be activated by default or its usage can be triggered by Sphinx tags.
 
-All collected and generated files get registered to the Sphinx Env and are therefore available during a Sphinx build.
+Depending on the usage of a specific collection for a build, its content integration can be controlled by the
+``if-collection::`` directive.
 
-It was created to support the following use cases:
+Following use cases are supported:
 
-* Grab additional ``.rst`` or ``md`` files from outside the ``docs`` source folder.
-* Merge multiple Sphinx projects into one project
-* Generate ``.rst`` and ``.md`` files based on data in ``json`` files.
+* Create file with content from string
+* Create file with content from function call
+* Copy single file from local path
+* Copy folder tree from local path
+* Create a symlink to a local target
+* Create a usage-report of collections
+* Clone git repository
+* Create multiple files based on jinja-template and specific data
 
-Internally ``Sphinx-Collections`` is based on a set of ``drivers``, which support different use cases.
-Feel free to extend the list of available ``drivers`` by creating a PR in our github project.
+``Sphinx-Collections`` cares about keeping your collection folders clean before and after each build.
 
 Introduction
 ------------
@@ -35,7 +49,7 @@ Introduction
    }
 
 The driver ``copy_folder`` allows to copy local folders and their files into your Sphinx project.
-There are other drivers available, which support different use cases and and files locations.
+There are other drivers available, which support different use cases and and file locations.
 
 By default all files get copied to ``_collections/`` + ``collection_name``, so in this example the complete path
 inside your documentation folder would be ``_collections/my_files/``. The location can be set specific for each

docs/conf.py

@@ -87,11 +87,50 @@ def clean(self):
         'driver': 'report',
         'target': 'doc_collection_report.rst',
         'active': True,
-    }
+    },
+    'symlink_test': {
+        'driver': 'symlink',
+        'source': '../tests/dummy/',
+        #'target': 'doc_collection_report.rst',
+        'active': True,
+    },
+    'jinja_test': {
+        'driver': 'jinja',
+        'source': 'examples/jinja_template.rst.temp',
+        'target': 'my_jinja_test_{{name}}.rst',
+        'data': {
+            'name': 'me',
+            'city': 'munich'
+            },
+        'active': True,
+    },
+    'jinja_test_multiple': {
+        'driver': 'jinja',
+        'source': 'examples/jinja_template.rst.temp',
+        'target': 'my_jinja_test_{{name|lower}}.rst',
+        'multiple_files': True,
+        'data': [
+            {
+            'name': 'Marco',
+            'city': 'Munich'
+            },
+            {
+            'name': 'Daniel',
+            'city': 'Soest'
+            },
+        ],
+        'active': True,
+    },
+    'git_test': {
+        'driver': 'git',
+        'source': 'https://github.com/useblocks/sphinx_dummy.git',
+        #'target': 'doc_collection_report.rst',
+        'active': True,
+    },
 
 }
 
-collections_final_clean = False
+collections_final_clean = True
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']

docs/drivers/copy_folder.rst

@@ -4,14 +4,13 @@ copy_folder
 Copies a folder tree from ``source`` into your documentation project::
 
     collections = {
-      'my_files: {
-         'driver': 'copy_folder',
-         'source': '../../extra_files/',
-         'target': 'my_data/'
-         'ignore': ['*.dat', '.exe'],
-         }
-      }
-   }
+        'my_files': {
+            'driver': 'copy_folder',
+            'source': '../../extra_files/',
+            'target': 'my_data/'
+            'ignore': ['*.dat', '.exe'],
+        }
+    }
 
 Options
 -------

docs/drivers/git.rst

@@ -0,0 +1,2 @@
+.. automodule:: sphinxcontrib.collections.drivers.git
+

docs/drivers/index.rst

@@ -11,16 +11,13 @@ Each collection must reference a single driver, which cares about:
 * Configured execution
 * Final clean up
 
-``Sphinx-Collections`` already provides some major drivers, which support different use case.
+``Sphinx-Collections`` already provides some major drivers, which support different use cases.
 
 .. toctree::
+   :glob:
    :maxdepth: 1
 
-   copy_folder
-   copy_file
-   string
-   function
-   report
+   *
 
 Own drivers
 -----------
@@ -51,7 +48,7 @@ Using own drivers instead of e.g. a pure function call has several advantages:
     register_driver('my_driver', myDriver)
 
     collections = {
-        'my_river_test': {
+        'my_driver_test': {
             'driver': 'my_driver',
             'source': '../tests/dummy/',
             'active': True,

docs/drivers/jinja.rst

@@ -0,0 +1,2 @@
+.. automodule:: sphinxcontrib.collections.drivers.jinja
+

docs/drivers/symlink.rst

@@ -0,0 +1,2 @@
+.. automodule:: sphinxcontrib.collections.drivers.symlink
+

docs/examples/jinja_template.rst.temp

@@ -0,0 +1,3 @@
+Hey Hooo!
+
+It's {{name}} from {{city}}!

setup.py

@@ -4,7 +4,7 @@
 import os
 from setuptools import setup, find_packages
 
-requires = ['sphinx>2.0', 'jinja2']
+requires = ['sphinx>2.0', 'jinja2', 'gitpython']
 
 setup(
     name='sphinx-collections',

sphinxcontrib/collections/collections.py

@@ -8,6 +8,9 @@
 from sphinxcontrib.collections.drivers.string import StringDriver
 from sphinxcontrib.collections.drivers.function import FunctionDriver
 from sphinxcontrib.collections.drivers.report import ReportDriver
+from sphinxcontrib.collections.drivers.symlink import SymlinkDriver
+from sphinxcontrib.collections.drivers.jinja import JinjaDriver
+from sphinxcontrib.collections.drivers.git import GitDriver
 
 sphinx_version = sphinx.__version__
 if parse_version(sphinx_version) >= parse_version("1.6"):
@@ -26,6 +29,9 @@
     'string': StringDriver,
     'function': FunctionDriver,
     'report': ReportDriver,
+    'symlink': SymlinkDriver,
+    'jinja': JinjaDriver,
+    'git': GitDriver,
 }
 
 
@@ -52,9 +58,6 @@ def execute_collections(app, config):
 
 
 def final_clean_collections(app, exception):
-    if not bool(app.config['collections_final_clean']):
-        return
-
     LOG.info('Final clean of collections ...')
     for collection in COLLECTIONS:
         collection.final_clean()
@@ -88,7 +91,7 @@ def __init__(self, app, name, **kwargs):
 
         # Check if we manipulate data only in documentation folder.
         # Any other location is not allowed.
-        if not os.path.realpath(target).startswith(os.path.realpath(app.confdir)):
+        if not target.startswith(os.path.realpath(app.confdir)):
             raise CollectionsException(
                 'Target path is not part of documentation folder\n'
                 'Target path: {}\n'
@@ -100,18 +103,19 @@ def __init__(self, app, name, **kwargs):
 
         self.target = target
 
-        clean = bool(kwargs.get('clean', True))
+        clean = kwargs.get('clean', None)
         if clean is None:
             clean = app.config['collections_clean']
         self.needs_clean = clean
 
-        final_clean = bool(kwargs.get('final_clean', True))
+        final_clean = kwargs.get('final_clean', None)
         if final_clean is None:
             final_clean = app.config['collections_final_clean']
         self.needs_final_clean = final_clean
 
         self.config = kwargs
         self.config['name'] = self.name
+        self.config['confdir'] = self.app.confdir
         self.config['target'] = target
         if 'safe' not in self.config.keys():
             self.config['safe'] = True

sphinxcontrib/collections/drivers/__init__.py

@@ -1,3 +1,4 @@
+import os
 import sphinx
 
 from pkg_resources import parse_version
@@ -82,6 +83,38 @@ def debug(self, message):
         """
         self._log.debug('{}{}'.format(self._prefix, message))
 
+    def get_source_path(self):
+        """
+        Returns absolute source path.
+        If source was configured as relative path, the absolute path is calculated
+        taking documentation confdir as base folder.
+
+        :return: path string
+        """
+        source = self.config.get('source', None)
+        if source is None:
+            self.error('Source must be defined')
+        if not os.path.isabs(source):
+            source = os.path.join(self.config['confdir'], source)
+        return source
+
+    def get_path(self, path):
+        """
+        Returns absolute path.
+        If path is given as relative path, the absolute path is calculated
+        taking documentation confdir as base folder.
+
+        :return: path string
+        """
+        if path is None:
+            self.error('Path must be defined')
+        if not isinstance(path, str):
+            self.debug('This functions makes mostly sense for string source only.')
+            return path
+        if not os.path.isabs(path):
+            path = os.path.join(self.config['confdir'], path)
+        return path
+
 
 class ColectionsDriverError(BaseException):
     pass

sphinxcontrib/collections/drivers/git.py

@@ -0,0 +1,53 @@
+"""
+git
+===
+
+``git`` driver clones a given repository into a target folder.
+
+URL must be given by ``source`` parameter.
+
+.. code-block:: python
+
+    collections = {
+        'my_files': {
+            'driver': 'git',
+            'source': 'https://github.com/useblocks/sphinx_dummy.git',
+        }
+    }
+
+``Sphinx-Collections`` will clone the given repository into ``_collections/my_files/``.
+
+.. hint::
+
+   ``Git`` binary must be installed on the system, so that the used Python library ``gitpython`` can use it.
+
+
+"""
+
+
+from shutil import rmtree
+
+from git import Repo
+from sphinxcontrib.collections.drivers import Driver
+
+
+class GitDriver(Driver):
+
+    def run(self):
+        self.info('Cloning git repository...')
+
+        try:
+            Repo.clone_from(self.config['source'],
+                            self.config['target'])
+        except Exception as e:
+            self.error('Problems during cloning repository.', e)
+
+    def clean(self):
+        try:
+            rmtree(self.config['target'])
+            self.info('Repository deleted: {}'.format(self.config['target']))
+        except FileNotFoundError:
+            # Already cleaned? I'm okay with it.
+            self.info('Cloned repository folder already cleaned.')
+        except IOError as e:
+            self.error('Problems during cleaning for collection {}'.format(self.config['name']), e)