Refactor caching, API (#634)

Sorry for this breaking change, but I wanted to get the foundations
right before tagging 1.0.

This completely restructures the core of citar so that now:

- the API in general is (again) citekey-based
- the note API is improved, and configured by plist
- the cache auto-invalidates when files change

See documentation for details.

As a result, we remove some redundant code:

- citar-filenotify.el
- the rebuild cache hook, etc.

There are also a number of code refinements.

Co-authored-by: Bruce D'Arcus <[email protected]>

.dir-locals.el

@@ -1,9 +1,11 @@
-((emacs-lisp-mode
-  (fill-column . 110)
-  (indent-tabs-mode . nil)
-  (elisp-lint-indent-specs . ((describe . 1)
-                              (it . 1)
-                              (thread-first . 0)
-                              (cl-flet . 1)
-                              (sentence-end-double-space . nil)
-                              (cl-flet* . 1)))))
+;;; Directory Local Variables
+;;; For more information see (info "(emacs) Directory Variables")
+
+((emacs-lisp-mode . ((sentence-end-double-space . nil)
+                     (fill-column . 110)
+                     (indent-tabs-mode . nil)
+                     (elisp-lint-indent-specs . ((describe . 1)
+                                                 (it . 1)
+                                                 (thread-first . 0)
+                                                 (cl-flet . 1)
+                                                 (cl-flet* . 1))))))

.github/workflows/check.yml

@@ -17,6 +17,7 @@ on:
 jobs:
   check:
     runs-on: ubuntu-latest
+    continue-on-error: ${{ matrix.emacs_version == 'snapshot' }}
     strategy:
       fail-fast: false
       matrix:
@@ -28,6 +29,9 @@ jobs:
           - compile
           - test
           - lint
+        exclude:
+          - emacs_version: snapshot
+            action: compile
     steps:
     - name: Set up Emacs ${{matrix.emacs_version}}
       uses: purcell/setup-emacs@master
@@ -63,5 +67,6 @@ jobs:
         eldev --color lint re
     - name: Lint package metadata
       if: ${{ matrix.action == 'lint' }}
+      continue-on-error: true
       run: |
-        eldev --color lint package || true
+        eldev --color lint package

CONTRIBUTING.org

@@ -5,31 +5,49 @@
 
 If you would like to contribute, details:
 
-- For more signifiant potential changes, file an issue first to get feedback on the basic idea.
+- For more significant potential changes, file an issue first to get feedback on the basic idea.
 - If you do submit a PR, follow the [[https://github.com/bbatsov/emacs-lisp-style-guide][elisp style guide]], and [[https://cbea.ms/git-commit/][these suggestions]] on git commit messages.
 - For working on lists and such, we primarily use the =seq= functions, and occassionally ~dolist~.
 
+** Basic Architecture
+
+Citar uses a cache, which stores two hash tables for each bibliography file:
+
+- entries ::  as returned by =parsebib-parse=, keys are citekeys, values are alists of entry fields
+- pre-formatted :: values are partially-formatted completion strings
+
+The =citar--ref-completion-table= function returns a hash table from the bibliographic cache, and ~citar--get-entry~ and ~-citar--get-value~ provide access to those data.
+Most user-accessible citar functions take an argument ~key~ or ~keys~.
+Some functions also take an ~entry~ argument, and ~citar--get-value~ takes either.
+When using these functions, you should keep in mind that unless you pass an entry alist to ~citar--get-value~, and instead use a key, each call to that function will query the cache.
+This, therefore, is a better pattern to use:
+
+#+begin_src emacs-lisp
+
+(let* ((entry (citar--get-entry key))
+       (title (citar--get-value entry "title")))
+  (message title))
+
+#+end_src
+
+
 ** Extending citar
 
-Most user-accessible citar functions take an argument ~key-entry~ or ~keys-entries~.
-These expect, respectively, a cons cell of a citation key (a string like "SmithWritingHistory1987") and the corresponding bibliography entry for that citation, or a list of such cons cells.
-If you wish to extend citar at the user-action level, perhaps by adding a function to one of the embark keymaps, you will find it easiest to reproduce this pattern.
-If you need to build the cons cells manually, this can be accomplished via ~citar--get-entry~.
-So, for example, to insert the annotations from a pdf into a buffer, the following pair of functions might be used:
+You can use ~citar-select-ref~ or ~citar-select-refs~ to write custom commands.
+An example:
 
 #+begin_src emacs-lisp
 
 
-(defun my/citar-insert-annots (keys-entries)
+(defun my/citar-insert-annots (keys)
   "insert annotations as org text from KEYS-ENTRIES"
-  (interactive (list (citar-select-refs
-                      :rebuild-cache current-prefix-arg)))
+  (interactive (list (citar-select-refs)))
   (let* ((files
-          (seq-mapcat (lambda (key-entry)
+          (seq-mapcat (lambda (key)
                         (citar-file--files-for-entry
-                         (car key-entry) (cdr key-entry)
+                         key (citar--get-entry key)
                          '("/") '("pdf")))
-                      keys-entries ))
+                      keys ))
          (output (seq-map
                   (lambda (file)
                     (pdf-annot-markups-as-org-text ;; you'll still need to write this function!
@@ -44,8 +62,7 @@ So, for example, to insert the annotations from a pdf into a buffer, the followi
 
   (defun my/independent-insert-annots (key)
     "helper function to insert annotations without the bibtex-actins apparatus"
-    (let ((key-entry (cons key  (citar--get-entry key))))
-      (my/citar-insert-annots (list key-entry))))
+    (my/citar-insert-annots (list key)))
 
 
 #+end_src

Eldev

@@ -8,18 +8,21 @@
 
 (eldev-use-plugin 'autoloads)
 
-(setf eldev-standard-excludes `(:or ,eldev-standard-excludes "./test/manual/" "./citar-capf.el"))
+;; TODO what to do with these excluded files?
+(setf eldev-standard-excludes `(:or ,eldev-standard-excludes "./test/manual/" "./citar-capf.el" "./citar-filenotify.el"))
 
 ;; (setf eldev-test-fileset '("./test/" "!./test/manual/"))
-;; (eldev-add-extra-dependencies 'test 'embark 'consult 'marginalia 'vertico 'auctex)
+(eldev-add-extra-dependencies '(build test lint) 'embark 'auctex)
 
 ;; allow to load test helpers
 ;; (eldev-add-loading-roots 'test "test/utils")
 
+;;; Linting settings
+
 ;; Tell checkdoc not to demand two spaces after a period.
 (setq sentence-end-double-space nil)
-
-(setf eldev-lint-default '(elisp))
+(setq eldev-lint-default '(elisp))
+(setq eldev-lint-stop-mode 'linter)
 
 (with-eval-after-load 'elisp-lint
   ;; Used eldev lint package | checkdoc
@@ -28,3 +31,7 @@
   ;; Emacs 29 snapshot has new indentation convention for cl-letf
   (when (> emacs-major-version 28)
     (push "indent" elisp-lint-ignored-validators)))
+
+;; Currently, package-lint has no other way of ignoring  checks.
+;; See https://github.com/purcell/package-lint/issues/125
+(advice-add #'package-lint--check-eval-after-load :override #'ignore)

README.org

@@ -17,8 +17,6 @@
    :CUSTOM_ID: features
    :END:
 
-Note: this package was formerly called "bibtex-actions."
-
 This package provides a completing-read front-end to browse and act on BibTeX, BibLaTeX, and CSL JSON bibliographic data, and LaTeX, markdown, and org-cite editing support.
 
 When used with vertico, embark, and marginalia, it provides similar functionality to helm-bibtex and ivy-bibtex: quick filtering and selecting of bibliographic entries from the minibuffer, and the option to run different commands against them.
@@ -50,6 +48,8 @@ In addition, the following packages are strongly recommended for the best experi
 3. [[https://github.com/oantolin/embark][Embark]] (contextual actions)
 4. [[https://github.com/minad/marginalia][Marginalia]] (annotations, and also candidate classification for Embark)
 
+We also recommend Emacs 28, as this package relies on two of its features, that greatly enhance the UI.
+
 ** Configuration
    :PROPERTIES:
    :CUSTOM_ID: configuration
@@ -73,9 +73,16 @@ This is the minimal configuration, and will work with any completing-read compli
 
 *** Embark
 
-Citar will automatically integrate with Embark if it is installed, offering contextual access to actions in the minibuffer and at-point.
+The =citar-embark= package adds contextual access actions in the minibuffer and at-point via the ~citar-embark-mode~ minor mode.
 When using Embark, the Citar actions are generic, and work the same across org, markdown, and latex modes.
 
+#+BEGIN_SRC emacs-lisp
+(use-package citar-embark
+  :after citar embark
+  :no-require
+  :config (citar-embark-mode))
+#+END_SRC
+
 *** Org-Cite
 
 #+CAPTION: org-cite at-point integration with =embark-act=
@@ -216,65 +223,17 @@ You can save this history across sessions by adding =citar-history= to =savehist
     :CUSTOM_ID: refreshing-the-library-display
     :END:
 
-=citar= uses two caches to speed up library display; one for the global bibliography, and another for local files specific to a buffer.
-This is great for performance, but means the data can become stale if you modify it.
-
-The =citar-refresh= command will reload the caches, and you can call this manually.
-You can also call any of the =citar= commands with a prefix argument: =C-u M-x citar-insert-key=.
-
-Although not default, =citar= also provides convenience functions for auto-refreshing cache when bib files change using filenotify.
-The simplest use of this functionality is
-
-#+BEGIN_SRC emacs-lisp
-(citar-filenotify-setup '(LaTeX-mode-hook org-mode-hook))
-#+END_SRC
-
-This will add watches for the global bib files and in addition add a hook to =LaTeX-mode-hook= and =org-mode-hook= to add watches for local bibliographic files.
-By default this will invalidate the cache if a bib file changes. If the bib files change rarely, a more suitable option is to refresh the cache.
-This can be achieved by
-
-#+BEGIN_SRC emacs-lisp
-(setq citar-filenotify-callback 'refresh-cache)
-#+END_SRC
-
-The behavior can be tweaked more thoroughly by setting ~citar-filenotify-callback~ to a function.
-See its documentation for details.
-Watches can be also placed on additional files.
-This is controlled by the variable ~citar-filenotify-files~.
-
-Another option to make the completion interface more seamless is to add a hook which generates the cache after a buffer is opened.
-This can be done when emacs has been idle (half a second in the example below) with something like this:
-
-#+BEGIN_SRC emacs-lisp
-(defun gen-bib-cache-idle ()
-  "Generate bib item caches with idle timer"
-  (run-with-idle-timer 0.5 nil #'citar-refresh))
-
-(add-hook 'LaTeX-mode-hook #'gen-bib-cache-idle)
-(add-hook 'org-mode-hook #'gen-bib-cache-idle)
-#+END_SRC
-
-For additional configuration options on this, see [[https://github.com/bdarcus/citar/wiki/Configuration#automating-path-watches][the wiki]].
+Citar uses a cache to speed up library display.
+If a bib file changes, the cache will automatically update the next time you run a Citar command.
+Note that cached data preformatted completion candidates are independently tracked by file.
+So, for example, if you have one very large bibliography file that changes a lot, you might consider splitting into one large file that is more stable, and one-or-more smaller ones that change more frequently.
 
 ** Notes
 
-Citar provides a ~citar-create-note-function~ variable, and a default function for org, which also works well with org-roam (v2 now supports org-cite).
-You can configure the title display using the "note" template.
-
-You can also use the ~citar-open-note-functions~ variable to replace or augment the default with another; for example from org-roam-bibtex:
-
-#+BEGIN_SRC emacs-lisp
-(setq citar-open-note-functions '(orb-citar-edit-note))
-#+END_SRC
-
-Since ~citar-open-note-functions~ is a list, you can also include multiple functions, to handle different note scenarios.
-
-Please note that if you choose to use org-roam-bibtex using the above configuration, you will need to set ~:immediate-finish t~ in the template that you use for bibliography notes in ~org-roam-capture-templates~.
-Since ~citar-open-note-functions~ attempts multiple functions one after the other, this is needed to ensure the ~org-capture~ returns immediately without waiting for further user input.
-
-Citar also includes a ~citar-has-notes-functions~ variable, which specifies a list of functions, each of which returns a predicate function to test whether a reference has associated notes.
-This function allows Citar to correctly format the completion UI candidates.
-The default function only supports one-file-per-key notes.
+Citar offers configurable note-taking and access integration.
+The ~citar-notes-sources~ variable configures note backends, and ~citar-notes-source~ activates your chosen backend.
+A backend primarily specifies functions to update the Citar display, to create the completion candidates, and to open existing and new notes.
+See the ~citar-notes-sources~ docstring for details, and the =citar-register-note-source= and =citar-remove-note-source= convenience functions.
 
 ** Files, file association and file-field parsing