crawling for fpm.toml files

[urbanjost]

Since at least github is searchable and fpm(1) is progressing rapidly I was thinking about a "listme" field in the fpm.toml file.

I have not figured out how to automate it yet, but github allows you to search in all repositories you have access to. So it is possible to look for all files called fpm.toml. Since there is another package that uses that filename it ends up all the hits are not all Fortran fpm files; but you can also access or search in the files. So you can at least crawl github and find fpm.toml files and inspect them to see if they look like Fortran fpm files. You can even query how many stars the repository has.

But the owners might just be trying something out and not have something they think is something that should be listed somewhere; or maybe I pulled someone else's repository. So I was thinking if you had an entry that said "listme" and a site name so you could match it to where it was found so you could tell a pull or a copy from an "original" that you could automate (assuming there is some CLI way to do this -- I have been clicking buttons) that you could automatically build a list of packages and put it on the fpm site. And you could even envision someone wanting to be a candidate in a reviewed repository being able to add a "review requested commit name" field as a second way of submitting your repository for addition to the reviewed repository list.

Anyone got a "why do that clicking, just use ????" idea? Anyone think this is a good, bad, or ugly idea? Think the idea is good but know a better way to crawl for the files?

Seems like a low-maintenance way to build up a list of fpm packages available (at least on github) that a developer can opt into (or out of with a "do not disturb" entry) with a really low entry barrier.

I am assuming this would generate an automated list on the fpm site. The metadata keys already supported could be used to categorize and annotate the listings. Timestamps would be useful too as to when released, when updated, ... but that is getting even further ahead.

[awvwgk]

Searching for https://github.com/search?p=1&q=fpm.toml&type=Code gave me only three false positives. Of course it searches the file content rather than the file names. We have the huge problem when searching for fpm due to the name collision with other three letter named projects (like effing package management, fingerprint module, fixed point math and potentially more).

There is other meta data which might be more useful, the package registry place to easily find fpm packages, but maybe one could search for usage of the setup-fpm action in GitHub CI workflows as well, once fpm finds wider adoption. Still it might be difficult to search also on Gitlab, Bitbucket, Sourceforge, or other less well known SCM hosting providers.

[urbanjost]

If I use github search in the upper left of the main webpage and do a site search for "filename:fpm.tom" I get dozens; but I have seen two sites mentioned recently that had fpm.toml files that did not show in the list, so I think there is an option to opt out of searches.

[awvwgk]

Using fpm.toml in:path improves the accuracy, I get the first false positives on page 6. Limiting the language to Fortran in the search might help here as well.

[everythingfunctional]

I think this is a reasonable approach - if it's not too difficult - as a stop gap solution. Eventually, we should have an official registry/hosting for released packages that can be published too. Hopefully doing so will be simple enough with something like fpm publish, or maybe fpm dist and uploading a generated tarball via a web interface. Basically the end goal for package publishing, distribution and use should mirror Cargo.

[milancurcic]

Can't fpm-registry be used for this?

[awvwgk]

Doing so makes me wonder why there has been few attempts to add packages to the registry.

For one missing documentation might be the main issue. The submission guidelines of the fpm-registry are literally the guidelines for submitting to the package index on fortran-lang.org (they even include a point on fpm-support).

[milancurcic]

There just aren't many fpm packages. fpm only recently started having release distributions. We had almost no outreach about fpm. It will take some time before people learn about it and start adapting their projects to fpm. It will take more time until the same people learn about fpm-registry and decide to submit their project. In short, we don't have traction yet. This is okay. Traction will come.

Yes, fpm-registry documentation is weak but I doubt that's an issue. A user that has an fpm-ready package has likely already passed the bar for having the skills to submit a PR to the repo.

[milancurcic]

Also, fpm-registry is currently not very useful in itself. It's just a list of packages and their URLs. It will become really useful when we can do something like fpm search <search-term> to list all available packages related to the search term.

[awvwgk]

Yes, fpm-registry documentation is weak but I doubt that's an issue. A user that has an fpm-ready package has likely already passed the bar for having the skills to submit a PR to the repo.

I've to disagree on this, I think it takes more than being able to open a PR on the registry for this. Without actual documentation neither the scope nor the expectation of a project are clear to a developer interested in submitting a package. At least for me this is part of the reason why I haven't considered submitting my fpm-enabled projects to the registry yet.

I opened fortran-lang/fpm-registry#27 on this topic.

[milancurcic]

At least for me this is part of the reason why I haven't considered submitting my fpm-enabled projects to the registry yet.

Okay, that's a fair and important point. Do you know of many other fpm packages that aren't listed in the fpm-registry and that satisfy the criteria for inclusion? I'm not aware of them.

[awvwgk]

I put together a list of all fpm projects on GitHub (+ some on GitLab) together with the timestamp when the fpm.toml file was originally added to the project:

https://github.com/fortran-lang/talks/blob/419107c/FortranCon2021-fpm/data/fpm-projects.txt

In total there are currently 173 projects with ~160 unique contributors over all fpm projects.

[everythingfunctional]

I think this is actually really impressive. fpm has barely existed for 18 months, let alone been ready for general use. This is a signal to me of both how desperately Fortran needed such a tool, and how great a job we've all done designing it. The success of this project just continues to amaze me. Oh, and it's written in Fortran, a language everybody keeps saying is only good for scientific computing!

[awvwgk]

The number of fpm project is also rising quite steadily, but we are not yet in an exponential growth:

[vmagnin]

That's more than respectable for a young software still considered in alpha phase!
The communication effort with that FortranCon 2021, maybe PackagingCon 2021, the CiSE article, etc. is needed to accelerate that evolution (revolution?). On my side, I have made some efforts to publicize fpm on the French web: for example, there is a paragraph in the French Fortran Wikipedia page (while I can not found it in the English Wikipedia page).