clojars/maven like pull mechanism for pods

[zcaudate]

it'd be great to have some sort of a repository for pods - maybe leveraging maven itself so that you can specify the dependency and have that dependency be pulled down, compiled (the first time) and loaded by bb.

[borkdude]

Are there any existing examples of systems that leverage maven for downloading binaries and running them? I think you first have to unzip the .jar to the filesystem before you can run a binary?

A system like brew seems more geared towards distributing binaries.

Note that pods can be implemented in any language, including Rust or Python. How would you declare a dependency on Python with Maven? With brew this is possible.

[zcaudate]

I'm not too familiar with how the bb.pods system works. I was following the honeysql example trying to get a couple of my libraries to work.

Not really knowing why it wasn't working, I guessed that I had to compile the libs into a pod in order to have it be usable with bb. That's why I thought about using maven to download the sources and then have them compiled because I assumed that in order to use a pod, it has to be compiled with graal in current system.

As I'm on OSX, I'm happy if the pods are being distributed via brew but I wouldn't know how they'll be distributed on other OS's. But whatever form bpm takes, I think it's needed to make writing and distributing pods easier.

[borkdude]

@zcaudate The problem with the hara.log require is that the namespace could not be found. This works:

borkdude@MBP2019 /tmp $ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {hara/log {:mvn/version "3.0.12"}}}')
Downloading: hara/log/3.0.12/log-3.0.12.pom from clojars
...
java.lang.Exception: Unable to resolve classname: java.util.concurrent.Executors [at hara/io/concurrent.clj, line 1, column 1]

You needed export to export the variable to the babashka process.
So the next problem is that babashka doesn't have the class java.util.concurrent.Executors yet. Considering the amount of dependencies, I don't expect that to be the last missing piece.

[borkdude]

Brew works for linux and macOS. For Windows a similar thing is called scoop.

[zcaudate]

@borkdude: I was getting that error with fish, switching to bash I'm getting:

bash-3.2$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {hara/log {:mvn/version "3.0.12"}}}')                                             
bash-3.2$  bb -e "(require '[hara.log :as log])"                                                                                                      
clojure.lang.ExceptionInfo: Could not resolve symbol: clojure.lang.APersistentMap [at hara/core/base/check.clj, line 23, column 18] 

Would it be better to compile it into a pod?

[borkdude]

@zcaudate I think it would be good if you could get a picture of all the missing classes and other items that are needed to make your library work with babashka. You can do that with a local checkout of babashka and then use clojure -A:main -e "(require '[hara.log :as log])" to run it, instead of the binary. If classes are missing, you can add them locally to babashka.impl.classes and then run again, until it works. If it's only a handful of things, it might be good to support them in babashka. If it requires drastic changes to babashka, pod might be a good alternative.

[zcaudate]

sweet. I'll give that a go.

[zcaudate]

I'm now getting this error here when running in the local directory

bash-3.2$ clojure -A:main -e "(require '[hara.log :as log])"
Syntax error (FileNotFoundException) compiling at (babashka/impl/bencode.clj:1:1).
Could not locate sci/impl/namespaces__init.class, sci/impl/namespaces.clj or sci/impl/namespaces.cljc on classpath.

Full report at:
/var/folders/rc/4nxjl26j50gffnkgm65ll8gr0000gp/T/clojure-10279402846106979821.edn
bash-3.2$ cat /var/folders/rc/4nxjl26j50gffnkgm65ll8gr0000gp/T/clojure-10279402846106979821.edn
{:clojure.main/message
 "Syntax error (FileNotFoundException) compiling at (babashka/impl/bencode.clj:1:1).\nCould not locate sci/impl/namespaces__init.class, sci/impl/names
paces.clj or sci/impl/namespaces.cljc on classpath.\n",
 :clojure.main/triage
 {:clojure.error/phase :compile-syntax-check,
  :clojure.error/line 1,
  :clojure.error/column 1,
  :clojure.error/source "bencode.clj",
  :clojure.error/path "babashka/impl/bencode.clj",
  :clojure.error/class java.io.FileNotFoundException,
  :clojure.error/cause
  "Could not locate sci/impl/namespaces__init.class, sci/impl/namespaces.clj or sci/impl/namespaces.cljc on classpath."},

[borkdude]

@zcaudate Here are some tips on getting started:

https://github.com/borkdude/babashka/blob/master/doc/dev.md

One thing to note: you need to clone with the --recursive option.

[zcaudate]

okay, I got it started and resolved a couple of issues but this error came up.

bash-3.2$ clojure -A:main -e "(require '[hara.log :as log])"
clojure.lang.ExceptionInfo: Unable to resolve classname: AbstractMethodError [at hara/core/component.clj, line 73, column 15]

AbstractMethodError should just resolve to java.lang.AbstractMethodError but it doesn't seem like it has an effect when I add it to the the classes.clj file

[borkdude]

@zcaudate Got it. You should also add it to :imports in babashka.main.clj.

[zcaudate]

@borkdude. Done. How about deftype?

clojure.lang.ExceptionInfo: Could not resolve symbol: deftype

[borkdude]

@zcaudate deftype is not supported yet in SCI, the interpreter used by babashka. See babashka/sci#345. I don't expect this will be implemented soon.

[borkdude]

@zcaudate An option could be to use :bb reader conditionals and replace the deftype by defrecord or something else that is supported in babashka.

[zcaudate]

What's the hurdle with implementing deftype? Would it be possible to alias deftype to compile as defrecord in sci?

Also, is there a more systematic way of doing this? Like generating an edn file and reading from that?
babashka/babashka@master...zcaudate:master

[zcaudate]

@borkdude: This works in bb. deftype.

user=> (defmacro deftype [name & more] `(defrecord ~name ~@more))
user=> (deftype Hello [state]) 
#'user/Hello
user=> (Hello. 1)
{:state 1}
user=> (.-state (Hello. 1))          
1

[borkdude]

The hurdle with deftype is the implementation of interfaces. Bb is running an interpreter and cannot emit JVM bytecode like Clojure can to construct implementations of interfaces. So I guess deftype would fail in a lot of cases if we alias it to defrecord.

defrecord only works for implementing protocols at the moment.

[zcaudate]

that code does contain interfaces so I guess pods is the way forward then.

would something like this work?
https://github.com/raydac/j-j-jvm

[borkdude]

Writing down some thoughts I had.

If checking in binaries into git wasn't an anti-pattern, I think Clojure git deps could be used to implement an ecosystem of binaries more or less like brew, but using deps.edn. The classpath can then be used as the binary path, a bit similar to a nix env. I would have a purpose for this (babashka pods). The git deps stuff could be made to work with GraalVM as well I think, so everything could be very fast after deps have been downloaded (but even without GraalVM, it's already fast due to classpath caching).

Maybe instead of checking in the binaries themselves, there could be links to releases in the source, with checksums, etc. But most of the gnarly dependency stuff could be handled by tda.

Implementing something like brew this way would only require a couple of tens of lines of code maybe.

There should be a couple of conventions like where to store the binaries in the source root categorized per operating system / architecture.

E.g. a repo could have src/my-package-descriptor.edn with {:linux {:link ... :checksum ...} :windows ...}. We could have a function (resolve-binary 'my.package) which will search for my-package-descriptor.edn and will fetch the binary to .git-libs/binaries/my/package/sha/the-binary if it isn't there already. If it's already there, we can immediately use that path and execute the binary.

[borkdude]

Closing this issue and starting a new one, because the discussion in this one got a little bit off topic.

[borkdude]

Clojars/maven pull mechanism is implemented here now: https://github.com/babashka/pod-registry

[zcaudate]

@borkdude: wow. congrats. this is so cool.

now I'm just waiting for you reimplement the clojure compiler in truffle to get jit/hotspot for bb.

I'm kinda half joking here but somehow I have a feeling that it'll probably happen sooner rather than later.

[borkdude]

@zcaudate Co-incidentally, I was just thinking about this, haha.