Merge branch 'dev/0_32/paths_validation' into test/ceedling_0_32_rc

docs/ReleaseNotes.md

@@ -67,6 +67,7 @@ There's more to be done, but Ceedling's documentation is more complete and accur
 ### Small Deal Highlights 🥉
 
 - Effort has been invested across the project to improve error messages, exception handling, and exit code processing. Noisy backtraces have been relegated to the verbosity level of DEBUG as <insert higher power> intended.
+- Logical ambiguity and functional bugs within `:paths` and `:files` configuration handling have been resolved along with updated documentation.
 - A variety of small improvements and fixes have been made throughout the plugin system and to many plugins.
 - The historically unwieldy `verbosity` command line task now comes in two flavors. The original recipe numeric parameterized version (e.g. `[4]`) exist as is. The new extra crispy recipe includes — funny enough — verbose task names `verbosity:silent`, `verbosity:errors`, `verbosity:complain`, `verbosity:normal`, `verbosity:obnoxious`, `verbosity:debug`. 
 - This release marks the beginning of the end for Rake as a backbone of Ceedling. Over many years it has become clear that Rake's design assumptions hamper building the sorts of features Ceedling's users want, Rake's command line structure creates a messy user experience for a full application built around it, and Rake's quirks cause maintenance challenges. Particularly for test suites, much of Ceedling's (invisible) dependence on Rake has been removed in this release. Much more remains to be done, including replicating some of the abilities Rake offers.
@@ -191,6 +192,18 @@ This release of Ceedling stripped the feature back to basics and largely rewrote
 
 While this new approach is not 100% foolproof, it is far more robust and far simpler than previous attempts. Other new Ceedling features should be able to address shortcomings in edge cases.
 
+### `:paths` and `:files` handling bug fixes and clarification
+
+Most project configurations are relatively simple, and Ceedling's features for collecting paths worked fine enough. However, bugs and ambiguities lurked. Further, insufficient validation left users resorting to old fashioned trial-and-error troubleshooting.
+
+Much glorious filepath and pathfile handling now abounds:
+
+* The purpose and use of `:paths` and `:files` has been clarified in both code and documentation. `:paths` are directory-oriented while `:files` are filepath-oriented.
+* [Documentation](CeedlingPacket.md) is now accurate and complete.
+* Path handling edge cases have been properly resolved (`./foo/bar` is the same as `foo/bar` but was not always processed as such).
+* Matching globs were advertised in the documentation (erroneously, incidentally) but lacked full programmatic support.
+* Ceedling now tells you if your matching patterns don't work. Unfortunately, all Ceedling can determine is if a particular pattern yielded 0 results.
+
 ### Plugin system improvements
 
 1. The plugin subsystem has incorporated logging to trace plugin activities at high verbosity levels.

lib/ceedling/configurator.rb

@@ -273,47 +273,55 @@ def eval_environment_variables(config)
   end
 
 
+  # Eval config path lists (convert strings to array of size 1) and handle any Ruby string replacement
   def eval_paths(config)
-    # [:plugins]:[load_paths] already handled
-
-    paths = [ # individual paths that don't follow convention processed below
-      config[:project][:build_root],
-      config[:release_build][:artifacts]]
+    # :plugins ↳ :load_paths already handled
 
-    eval_path_list( paths )
+    eval_path_entries( config[:project][:build_root] )
+    eval_path_entries( config[:release_build][:artifacts] )
 
-    config[:paths].each_pair { |collection, paths| eval_path_list( paths ) }
+    config[:paths].each_pair do |entry, paths|
+      # :paths sub-entries (e.g. :test) could be a single string -> make array
+      reform_path_entries_as_lists( config[:paths], entry, paths )
+      eval_path_entries( paths )
+    end
 
-    config[:files].each_pair { |collection, files| eval_path_list( files ) }
+    config[:files].each_pair do |entry, files|
+      # :files sub-entries (e.g. :test) could be a single string -> make array
+      reform_path_entries_as_lists( config[:files], entry, files )
+      eval_path_entries( files )
+    end
 
-    # all other paths at secondary hash key level processed by convention:
-    # ex. [:toplevel][:foo_path] & [:toplevel][:bar_paths] are evaluated
-    config.each_pair { |parent, child| eval_path_list( collect_path_list( child ) ) }
+    # All other paths at secondary hash key level processed by convention (`_path`):
+    # ex. :toplevel ↳ :foo_path & :toplevel ↳ :bar_paths are evaluated
+    config.each_pair { |_, child| eval_path_entries( collect_path_list( child ) ) }
   end
 
 
   def standardize_paths(config)
     # [:plugins]:[load_paths] already handled
 
-    paths = [ # individual paths that don't follow convention processed below
+    # Individual paths that don't follow convention processed here
+    paths = [
       config[:project][:build_root],
-      config[:release_build][:artifacts]] # cmock path in case it was explicitly set in config
+      config[:release_build][:artifacts]
+    ]
 
     paths.flatten.each { |path| FilePathUtils::standardize( path ) }
 
     config[:paths].each_pair do |collection, paths|
-      # ensure that list is an array (i.e. handle case of list being a single string,
+      # Ensure that list is an array (i.e. handle case of list being a single string,
       # or a multidimensional array)
       config[:paths][collection] = [paths].flatten.map{|path| FilePathUtils::standardize( path )}
     end
 
-    config[:files].each_pair { |collection, files| files.each{ |path| FilePathUtils::standardize( path ) } }
+    config[:files].each_pair { |_, files| files.each{ |path| FilePathUtils::standardize( path ) } }
 
-    config[:tools].each_pair { |tool, config| FilePathUtils::standardize( config[:executable] ) if (config.include? :executable) }
+    config[:tools].each_pair { |_, config| FilePathUtils::standardize( config[:executable] ) if (config.include? :executable) }
 
-    # all other paths at secondary hash key level processed by convention:
-    # ex. [:toplevel][:foo_path] & [:toplevel][:bar_paths] are standardized
-    config.each_pair do |parent, child|
+    # All other paths at secondary hash key level processed by convention (`_path`):
+    # ex. :toplevel ↳ :foo_path & :toplevel ↳ :bar_paths are standardized
+    config.each_pair do |_, child|
       collect_path_list( child ).each { |path| FilePathUtils::standardize( path ) }
     end
   end
@@ -425,18 +433,27 @@ def insert_rake_plugins(plugins)
 
   private
 
+  def reform_path_entries_as_lists( container, entry, value )
+    container[entry] = [value]  if value.kind_of?( String )
+  end
+
   def collect_path_list( container )
     paths = []
     container.each_key { |key| paths << container[key] if (key.to_s =~ /_path(s)?$/) } if (container.class == Hash)
     return paths.flatten
   end
 
-  def eval_path_list( paths )
-    if paths.kind_of?(Array)
-      paths = Array.new(paths)
+  def eval_path_entries( container )
+    paths = []
+
+    case(container)
+    when Array then paths = Array.new( container ).flatten()
+    when String then paths << container
+    else
+      return
     end
 
-    paths.flatten.each do |path|
+    paths.each do |path|
       path.replace( @system_wrapper.module_eval( path ) ) if (path =~ RUBY_STRING_REPLACEMENT_PATTERN)
     end
   end

lib/ceedling/configurator_builder.rb

@@ -8,7 +8,7 @@
 
 class ConfiguratorBuilder
 
-  constructor :file_system_utils, :file_wrapper, :system_wrapper
+  constructor :file_path_collection_utils, :file_wrapper, :system_wrapper
 
 
   def build_global_constant(elem, value)
@@ -248,9 +248,9 @@ def expand_all_path_globs(in_hash)
       path_keys << key
     end
 
-    # sorted to provide assured order of traversal in test calls on mocks
-    path_keys.sort.each do |key|
-      out_hash["collection_#{key}".to_sym] = @file_system_utils.collect_paths( in_hash[key] )
+    path_keys.each do |key|
+      _collection = "collection_#{key}".to_sym
+      out_hash[_collection] = @file_path_collection_utils.collect_paths( in_hash[key] )
     end
 
     return out_hash
@@ -261,7 +261,7 @@ def collect_source_and_include_paths(in_hash)
     return {
       :collection_paths_source_and_include =>
         ( in_hash[:collection_paths_source] +
-          in_hash[:collection_paths_include] ).select {|x| File.directory?(x)}
+          in_hash[:collection_paths_include] )
       }
   end
 
@@ -281,10 +281,10 @@ def collect_source_include_vendor_paths(in_hash)
   def collect_test_support_source_include_paths(in_hash)
     return {
       :collection_paths_test_support_source_include =>
-        (in_hash[:collection_paths_test] +
-        in_hash[:collection_paths_support] +
-        in_hash[:collection_paths_source] +
-        in_hash[:collection_paths_include] ).select {|x| File.directory?(x)}
+        ( in_hash[:collection_paths_test] +
+          in_hash[:collection_paths_support] +
+          in_hash[:collection_paths_source] +
+          in_hash[:collection_paths_include] )
       }
   end
 
@@ -310,7 +310,7 @@ def collect_tests(in_hash)
       all_tests.include( File.join(path, "#{in_hash[:project_test_file_prefix]}*#{in_hash[:extension_source]}") )
     end
 
-    @file_system_utils.revise_file_list( all_tests, in_hash[:files_test] )
+    @file_path_collection_utils.revise_filelist( all_tests, in_hash[:files_test] )
 
     return {:collection_all_tests => all_tests}
   end
@@ -332,7 +332,7 @@ def collect_assembly(in_hash)
     end
 
     # Also add files that we are explicitly adding via :files:assembly: section
-    @file_system_utils.revise_file_list( all_assembly, in_hash[:files_assembly] )
+    @file_path_collection_utils.revise_filelist( all_assembly, in_hash[:files_assembly] )
 
     return {:collection_all_assembly => all_assembly}
   end
@@ -342,14 +342,10 @@ def collect_source(in_hash)
     all_source = @file_wrapper.instantiate_file_list
 
     in_hash[:collection_paths_source].each do |path|
-      if File.exist?(path) and not File.directory?(path)
-        all_source.include( path )
-      elsif File.directory?(path)
-        all_source.include( File.join(path, "*#{in_hash[:extension_source]}") )
-      end
+      all_source.include( File.join(path, "*#{in_hash[:extension_source]}") )
     end
 
-    @file_system_utils.revise_file_list( all_source, in_hash[:files_source] )
+    @file_path_collection_utils.revise_filelist( all_source, in_hash[:files_source] )
 
     return {:collection_all_source => all_source}
   end
@@ -367,7 +363,7 @@ def collect_headers(in_hash)
       all_headers.include( File.join(path, "*#{in_hash[:extension_header]}") )
     end
 
-    @file_system_utils.revise_file_list( all_headers, in_hash[:files_include] )
+    @file_path_collection_utils.revise_filelist( all_headers, in_hash[:files_include] )
 
     return {:collection_all_headers => all_headers}
   end
@@ -386,16 +382,12 @@ def collect_release_build_input(in_hash)
 
     # Collect source files
     in_hash[:collection_paths_source].each do |path|
-      if File.exist?(path) and not File.directory?(path)
-        release_input.include( path )
-      elsif File.directory?(path)
-        release_input.include( File.join(path, "*#{in_hash[:extension_source]}") )
-        release_input.include( File.join(path, "*#{in_hash[:extension_assembly]}") ) if in_hash[:release_build_use_assembly]
-      end
+      release_input.include( File.join(path, "*#{in_hash[:extension_source]}") )
+      release_input.include( File.join(path, "*#{in_hash[:extension_assembly]}") ) if in_hash[:release_build_use_assembly]
     end
 
-    @file_system_utils.revise_file_list( release_input, in_hash[:files_source] )
-    @file_system_utils.revise_file_list( release_input, in_hash[:files_assembly] ) if in_hash[:release_build_use_assembly]
+    @file_path_collection_utils.revise_filelist( release_input, in_hash[:files_source] )
+    @file_path_collection_utils.revise_filelist( release_input, in_hash[:files_assembly] ) if in_hash[:release_build_use_assembly]
 
     return {:collection_release_build_input => release_input}
   end
@@ -427,10 +419,10 @@ def collect_existing_test_build_input(in_hash)
       all_input.include( File.join(path, "*#{in_hash[:extension_assembly]}") ) if in_hash[:test_build_use_assembly]
     end
 
-    @file_system_utils.revise_file_list( all_input, in_hash[:files_test] )
-    @file_system_utils.revise_file_list( all_input, in_hash[:files_support] )
-    @file_system_utils.revise_file_list( all_input, in_hash[:files_source] )
-    @file_system_utils.revise_file_list( all_input, in_hash[:files_assembly] ) if in_hash[:test_build_use_assembly]
+    @file_path_collection_utils.revise_filelist( all_input, in_hash[:files_test] )
+    @file_path_collection_utils.revise_filelist( all_input, in_hash[:files_support] )
+    @file_path_collection_utils.revise_filelist( all_input, in_hash[:files_source] )
+    @file_path_collection_utils.revise_filelist( all_input, in_hash[:files_assembly] ) if in_hash[:test_build_use_assembly]
 
     return {:collection_existing_test_build_input => all_input}
   end
@@ -458,7 +450,7 @@ def collect_test_fixture_extra_link_objects(in_hash)
       support.include( File.join(path, "*#{in_hash[:extension_assembly]}") ) if in_hash[:test_build_use_assembly]
     end
 
-    @file_system_utils.revise_file_list( support, in_hash[:files_support] )
+    @file_path_collection_utils.revise_filelist( support, in_hash[:files_support] )
 
     # Ensure FileList patterns & revisions are resolved into full list of filepaths
     support.resolve()

lib/ceedling/configurator_setup.rb

@@ -74,28 +74,33 @@ def validate_required_section_values(config)
   end
 
   def validate_paths(config)
-    validation = []
+    valid = true
 
     if config[:cmock][:unity_helper]
       config[:cmock][:unity_helper].each do |path|
-        validation << @configurator_validator.validate_filepath_simple( path, :cmock, :unity_helper ) 
+        valid &= @configurator_validator.validate_filepath_simple( path, :cmock, :unity_helper ) 
       end
     end
 
     config[:project][:options_paths].each do |path|
-      validation << @configurator_validator.validate_filepath_simple( path, :project, :options_paths )
+      valid &= @configurator_validator.validate_filepath_simple( path, :project, :options_paths )
     end
 
     config[:plugins][:load_paths].each do |path|
-      validation << @configurator_validator.validate_filepath_simple( path, :plugins, :load_paths )
+      valid &= @configurator_validator.validate_filepath_simple( path, :plugins, :load_paths )
     end
 
     config[:paths].keys.sort.each do |key|
-      validation << @configurator_validator.validate_path_list(config, :paths, key)
+      valid &= @configurator_validator.validate_path_list(config, :paths, key)
+      valid &= @configurator_validator.validate_paths_entries(config, key)
     end
 
-    return false if (validation.include?(false))
-    return true
+    config[:files].keys.sort.each do |key|
+      valid &= @configurator_validator.validate_path_list(config, :files, key)
+      valid &= @configurator_validator.validate_files_entries(config, key)
+    end
+
+    return valid
   end
 
   def validate_tools(config)
@@ -109,7 +114,7 @@ def validate_tools(config)
   end
 
   def validate_threads(config)
-    validate = true
+    valid = true
 
     compile_threads = config[:project][:compile_threads]
     test_threads = config[:project][:test_threads]
@@ -118,35 +123,35 @@ def validate_threads(config)
     when Integer
       if compile_threads < 1
         @stream_wrapper.stderr_puts("ERROR: [:project][:compile_threads] must be greater than 0")
-        validate = false
+        valid = false
       end
     when Symbol
       if compile_threads != :auto
         @stream_wrapper.stderr_puts("ERROR: [:project][:compile_threads] is neither an integer nor :auto") 
-        validate = false
+        valid = false
       end
     else
       @stream_wrapper.stderr_puts("ERROR: [:project][:compile_threads] is neither an integer nor :auto") 
-      validate = false
+      valid = false
     end
 
     case test_threads
     when Integer
       if test_threads < 1
         @stream_wrapper.stderr_puts("ERROR: [:project][:test_threads] must be greater than 0")
-        validate = false
+        valid = false
       end
     when Symbol
       if test_threads != :auto
         @stream_wrapper.stderr_puts("ERROR: [:project][:test_threads] is neither an integer nor :auto") 
-        validate = false
+        valid = false
       end
     else
       @stream_wrapper.stderr_puts("ERROR: [:project][:test_threads] is neither an integer nor :auto") 
-      validate = false
+      valid = false
     end
 
-    return validate
+    return valid
   end
 
   def validate_plugins(config)