Integrate Algolia search indexing (#48)

* Add basic Algolia index push capabilities (#46) * Avoid overwriting data arrays when nesting * Fix param validation * Fix bug that held caused 'SystemStackError' Fixes 'SystemStackError thrown. stack level too deep' caused when by feeding the input variable into a nested hash. * Add key acceptance to CLI, clean up algolia_index_cli * Add docs for Algolia to README * Add LD config instructions to README * Add search: block to options table in README * Correct data nesting
DocOps · Aug 7, 2018 · c6ba5d4 · c6ba5d4
1 parent 17edd7b
commit c6ba5d4
Show file tree

Hide file tree

Showing 2 changed files with 151 additions and 19 deletions.
diff --git a/README.adoc b/README.adoc
@@ -1,4 +1,6 @@
 = LiquiDoc
+:toc: preamble
+
 // tag::overview[]
 LiquiDoc is a documentation build utility for true single-sourcing of technical content and data.
 It is especially suited for documentation projects with various required output formats, but it is intended for any project with complex, versioned input data for use in docs, user interfaces, and even back-end code.
@@ -877,15 +879,83 @@ See <<per-build-properties-files>>.
 
 === Deploy Operations
 
-It's not clear how deeply we will delve into deploy operations, since other build systems (such as rake) would seem far more suitable.
-For testing purposes, however, spinning up a local webserver with the same stroke that you build a site is pretty rewarding and time saving.
+Mainstream deployment platforms are probebly better suited to tying all your operations together, but we plan to bake a few common operations in to help you get started.
+For true build-and-deployment control, consider build tools such as Make, Rake, and Gradle, or deployment tools like Travis CI, CircleCI, and Jenkins.
+
+==== Jekyll Serve
+
+For testing purposes, however, spinning up a local webserver with the same stroke that you build a site is pretty rewarding and time saving, so we'll start there.
 
 For now, this functionality is limited to adding a `--deploy` flag to your `liquidoc` command.
 This will attempt to serve files from the *destination* set for the associated Jekyll build.
 
 [WARNING]
 Deployment of Jekyll sites is both limited and untested under nonstandard conditions.
 
+==== Algolia Search Indexing for Jekyll
+
+If you're using Jekyll to build sites, LiquiDoc makes indexing your files with the Algolia cloud search service a matter of configuration.
+The heavy lifting is performed by the jekyll-algolia plugin, but LiquiDoc can handle indexing even a complex site by using the same configuration that built your HTML content (which is what Algolia actually indexes).
+
+[NOTE]
+You will need a free community (or premium) link:https://www.algolia.com/users/sign_up/hacker[Algolia account] to take advantage of Algolia's indexing service and REST API.
+Simply create a named index, then visit the API Keys to collect the rest of the info you'll need to get going.
+
+Two hard-coding steps are required to prep your source to handle Algolia index pushes.
+
+. Add a block to your main Jekyll configuration file.
++
+.Example Jekyll Algolia configuration
+[source,yaml]
+----
+algolia:
+  application_id: 'your-application-id' # <1>
+  search_only_api_key: 'your-search-only-api-key' # <2>
+  extensions_to_index: [adoc] # <3>
+----
++
+<1> From the top bar of your Algolia interface.
+<2> From the API Keys screen of your Algolia interface.
+<3> List as many extensions as apply, separated by commas.
+
+. Add a block to your build config.
++
+[source,yaml]
+----
+  - action: render
+    data: globals.yml
+    builds:
+      - backend: jekyll
+        properties:
+          files:
+            - _configs/jekyll-global.yml
+            - _configs/jekyll-portal-1.yml
+          arguments:
+            destination: build/site/user-basic
+        attributes:
+          portal_term: Guide
+        search:
+          index: 'portal-1'
+----
++
+The `index:` parameter is for the name of the index you are pushing to.
+(An Algolia “app” can have multiple “indices”.)
+If you have
+
+Now you can call your same LiquiDoc build command with the `--search-index-push` or `--search-index-dry` flags along with the `--search-api-key='your-admin-api-key-here'` argument in order to invoke the indexing operation.
+The `--search-index-dry` flag merely tests content packaging, whereas `--search-index-push` connects to the Algolia REST API and attempt to push your content for indexing and storage.
+
+.Example Jekyll Algolia deployment
+[source,shell]
+----
+bundle exec liquidoc -c _configs/build-docs.yml --search-index-push --search-index-api-key='90f556qaa456abh6j3w7e8c10t48c2i57'
+----
+
+This operation performs a complete build, including each render operation, before the Algolia plugin processes content and pushes each build to the indexing service, in turn.
+
+[TIP]
+To add modern site search for your users, add link:https://community.algolia.com/instantsearch.js/[Algolia's InstantSearch functionality] to your front end!
+
 === Config Settings Matrix
 
 Here is a table of all the established configuration settings, as they pertain to each key LiquiDoc action.
@@ -992,6 +1062,12 @@ s| properties
 | N/A
 | Optional
 |
+
+s| search
+| N/A
+| N/A
+| Optional
+|
 |===
 
 pass:[*]The `output` setting is considered optional for render operations because static site generations target a directory set in the SSG's config file.

diff --git a/lib/liquidoc.rb b/lib/liquidoc.rb
@@ -1,7 +1,7 @@
 require 'liquidoc'
+require 'optparse'
 require 'yaml'
 require 'json'
-require 'optparse'
 require 'liquid'
 require 'asciidoctor'
 require 'asciidoctor-pdf'
@@ -48,6 +48,8 @@
 @verbose = false
 @quiet = false
 @explicit = false
+@search_index = false
+@search_index_dry = ''
 
 # Instantiate the main Logger object, which is always running
 @logger = Logger.new(STDOUT)
@@ -276,7 +278,7 @@ def message
             text = ". #{stage}Draws data from `#{self.data[0]}`"
           end
         else
-          text = ". #{stage}Draws data from `#{self.data['file']}`"
+          text = ". #{stage}Draws data from `#{self.data}`"
         end
         text.concat("#{reason},") if reason
         text.concat(" and parses it as follows:")
@@ -422,6 +424,18 @@ def prop_files_array
   #   props['files'].force_array if props['files']
   # end
 
+  def search
+    props['search']
+  end
+
+  def add_search_prop! prop
+    begin
+      self.search.merge!prop
+    rescue
+      raise "PropertyInsertionError"
+    end
+  end
+
   # NOTE this section repeats in Class.AsciiDocument
   def attributes
     @build['attributes']
@@ -586,11 +600,7 @@ def get_data datasrc
 
 # Pull in a semi-structured data file, converting contents to a Ruby hash
 def ingest_data datasrc
-# Must be passed a proper data object (there must be a better way to validate arg datatypes)
-  unless datasrc.is_a? Object
-    raise "InvalidDataObject"
-  end
-  # This proc should really begin here, once the datasrc object is in order
+  raise "InvalidDataObject" unless datasrc.is_a? Object
   case datasrc.type
   when "yml"
     begin
@@ -654,7 +664,7 @@ def parse_regex data_file, pattern
         end
       end
     end
-    output = {"data" => records}
+    output = records
   rescue Exception => ex
     @logger.error "Something went wrong trying to parse the free-form file. #{ex.class} thrown. #{ex.message}"
     raise "Freeform parse error"
@@ -665,8 +675,10 @@ def parse_regex data_file, pattern
 # Parse given data using given template, generating given output
 def liquify datasrc, template_file, output, variables=nil
   input = get_data(datasrc)
-  nested = { "data" => get_data(datasrc)}
-  input.merge!nested
+  unless input['data']
+    nested = { "data" => input.dup }
+    input.merge!nested
+  end
   validate_file_input(template_file, "template")
   if variables
     vars = { "vars" => variables }
@@ -878,8 +890,8 @@ def generate_site doc, build
   when "jekyll"
     attrs = doc.attributes
     build.add_config_file("_config.yml") unless build.prop_files_array
-    jekyll_config = YAML.load_file(build.prop_files_array[0]) # load the first Jekyll config file locally
-    attrs.merge! ({"base_dir" => jekyll_config['source']}) # Sets default Asciidoctor base_dir to == Jekyll root
+    jekyll = load_jekyll_data(build) # load the first Jekyll config file locally
+    attrs.merge! ({"base_dir" => jekyll['source']}) # Sets default Asciidoctor base_dir to == Jekyll root
     # write all AsciiDoc attributes to a config file for Jekyll to ingest
     attrs.merge!(build.attributes) if build.attributes
     attrs = {"asciidoctor" => {"attributes" => attrs} }
@@ -892,14 +904,31 @@ def generate_site doc, build
     if build.props['arguments']
       opts_args = build.props['arguments'].to_opts_args
     end
-    command = "bundle exec jekyll build --config #{config_list} #{opts_args} #{quiet}"
+    base_args = "--config #{config_list} #{opts_args}"
+    command = "bundle exec jekyll build #{base_args} #{quiet}"
+    if @search_index
+      # TODO enable config-based admin api key ingest once config is dynamic
+      command = algolia_index_cmd(build, @search_api_key, base_args)
+      @logger.warn "Search indexing failed." unless command
+    end
+  end
+  if command
+    @logger.info "Running #{command}"
+    @logger.debug "AsciiDoc attributes: #{doc.attributes.to_yaml} "
+    system command
   end
-  @logger.info "Running #{command}"
-  @logger.debug "AsciiDoc attributes: #{doc.attributes.to_yaml} "
-  system command
   jekyll_serve(build) if @jekyll_serve
 end
 
+def load_jekyll_data build
+  data = {}
+  build.prop_files_array.each do |file|
+    settings = YAML.load_file(file)
+    data.merge!settings if settings
+  end
+  return data
+end
+
 # ===
 # DEPLOY procs
 # ===
@@ -914,6 +943,20 @@ def jekyll_serve build
   system command
 end
 
+def algolia_index_cmd build, apikey=nil, args
+  unless build.search and build.search['index']
+    @logger.warn "No index configuration found for build; jekyll-algolia operation skipped for this build."
+    return false
+  else
+    unless apikey
+      @logger.warn "No Algolia admin API key passed; skipping jekyll-algolia operation for this build."
+      return false
+    else
+      return "ALGOLIA_INDEX_NAME='#{build.search['index']}' ALGOLIA_API_KEY='#{apikey}' bundle exec jekyll algolia #{@search_index_dry} #{args} "
+    end
+  end
+end
+
 # ===
 # Text manipulation Classes, Modules, procs, etc
 # ===
@@ -1081,7 +1124,7 @@ def regexreplace input, regex, replacement=''
     @quiet = true
   end
 
-  opts.on("--explicit", "Log explicit step descriptions to console as build progresses. (Otherwise writes to file at #{@build_dir}/pre/config-explainer.adoc .)") do |n|
+  opts.on("--explain", "Log explicit step descriptions to console as build progresses. (Otherwise writes to file at #{@build_dir}/pre/config-explainer.adoc .)") do |n|
     explainer_init("STDOUT")
     @explainer.level = Logger::INFO
     @logger.level = Logger::WARN # Suppress all those INFO-level messages
@@ -1096,6 +1139,19 @@ def regexreplace input, regex, replacement=''
     @jekyll_serve = true
   end
 
+  opts.on("--search-index-push", "Runs any search indexing configured in the build step and pushes to Algolia.") do
+    @search_index = true
+  end
+
+  opts.on("--search-index-dry", "Runs any search indexing configured in the build step but does NOT push to Algolia.") do
+    @search_index = true
+    @search_index_dry = "--dry-run"
+  end
+
+  opts.on("--search-api-key=STRING", "Passes Algolia Admin API key (which you should keep out of Git).") do |n|
+    @search_api_key = n
+  end
+
   opts.on("-h", "--help", "Returns help.") do
     puts opts
     exit