How to generate dynamic pages from other pages and shared data?

[boeckMt]

Maybe related to #303 (comment)

I have a site (which I'm currently trying to convert to Lume) with lots of subpages.

E.g. Folder structure

└─ site
    ├─ items/s1.md
    ├─ items/s2.md
    ├─ ...

I want to collect all these subpages and use it as Shared data so it can be reused on other sites as well.
E.g. create a overview list (like site activations) which lists all subsites (e.g site activations/id) and parts of there data.

I also want to create other subsites from this data dynamically e.g. site items/id/compare site activations/id/compare
I tried to use the example from Create multiple pages

// comparePages.tmpl.js
import lume from "lume/mod.ts";

export default function* () {
  const languages = ["de", "en"];
  const site = lume();

  // this list should be get from the Shared data
  const compareProducts = site.data.collections.filter(...); // [{ p_id: 'ACT006/P01' }, { p_id: 'ACT006/P02' }, { p_id: 'ACT006/P03' }]; 

  for (const lang of languages) {
    for (const product of compareProducts) {
      const a_p_id = product.p_id.split("/");
      const a_id = a_p_id[0];
      const p_num = a_p_id[1];

      yield {
        title: `Compare Test Site`,
        layout: "test-compare.pug",
        content: "",
        lang: lang,
        url: `/${lang}/activations/items/${a_id}/${p_num}/compare.html`,
      };
    }
  }
}

but I don't know how to reuse Shared data in .tmpl.js

I also tried using the search function but without success because it then looks as if shared data is not available in the templates.

export default function* ({ search }) {
  const languages = ["de", "en"];

  const compareProducts = search.pages("type=product")
  ...
}

---

In order to transfer the data to Shared Data, I currently created a plugin, but I don't know if this is the correct approach

/**
  site.use(collections({
  activations: {
    pattern: ["/activations/items/"],
    sortBy: "a_date"
  },
  products: {
    pattern: ["/products/items/"],
    sortBy: "p_date"
  },
}));
*/
export function collections(options: CollectionsOptions) {
  return (site: Site) => {
    const data: Collections = {};

    Object.keys(options).forEach((c) => {
      data[c] = [];
    });

    site.preprocess([".md", ".json"], (page) => {
      const path = page.src.path;

      Object.keys(options).forEach((c) => {
        const option = options[c];
        const paterns = option.pattern.map((p) => p);
        paterns.forEach((p) => {
          const isPath = path.includes(p);

          if (isPath) {
            const newData = Object.assign({}, page.data);
            const hasPage = data[c].find((p) => p.url === newData.url);
            if (!hasPage) {
              data[c].push(newData);
            }
          }
        });

      });
    });

    site.data("collections", data);
  };
}

[oscarotero]

Shared data is merged with the page data and passed to the first argument of the default function:

export default function* (data) {
    const { search } = data;

    const compareProducts = search.pages("type=product");

    // Create pages to compare the products
    for (const page1 of compareProducts) {
        for (const page2 of compareProducts) {
            if (page1 !== page2) {
              yield {
                 title: `Comparing ${page1.data.title} vs ${page2.data.title}`,
                 layout: "compare-page.njk",
                 page1,
                 page2,
              }
            }
        }
    }
}

[boeckMt]

Hey @oscarotero thanks for the response, you are really fast :)

Retrieving the pages with search and generating the sites works, but in the layout I cannot access the shared data.
I have some config in a config.yml

└─ site
    ├─ _data
        ├─ config.yml
    ├─ items/s1.md
    ├─ items/s2.md
    ├─ ...

Now when I want to retrieve this configuration in the layout file

// njk
<p>{{ config.key }}</p>

or

// pug
p
   =config.key

in njk there is no error but p is empty. In pug it says Cannot read properties of undefined (reading 'key')

If I use the same templates on other pages, not created with a generator function it works. Do I have to pass the data to yield {...} somehow?

[oscarotero]

The pages generated dynamically don't inherit the contextual data.
You need to include the desired data in the yielded object. For example:

export default function* (data) {
    const { search, config } = data;

    const compareProducts = search.pages("type=product");

    // Create pages to compare the products
    for (const page1 of compareProducts) {
        for (const page2 of compareProducts) {
            if (page1 !== page2) {
              yield {
                 title: `Comparing ${page1.data.title} vs ${page2.data.title}`,
                 layout: "compare-page.njk",
                 page1,
                 page2,
                 config,
              }
            }
        }
    }
}

[boeckMt]

@oscarotero thanks your response and clarification.
Maybe I can help add things like this more clearly to your documentation?

[oscarotero]

mm, maybe I spoke too fast.
The data of the generated pages are result of merging the main page with the yielded object. If it's not correct, probably there's a bug there?

[boeckMt]

I get two different objects when I inspect the page data in the template generator function and in site.preprocess.

// template to generate pages - comparePages.tmpl.ts
import { PageData } from "lume/core.ts";

export default function* (data: PageData) {

// here the config from site/_data/config.yml is missing 
// but data set with `site.data("collections", data);` in `site.use...` is included.


}

In the template generator function the config is missing, but in the following preprocess function it is available

// lume _config.ts

...
const site = lume({
  src: "./src",
  dest: "./dist/build",
  emptyDest: false,
  location: new URL(publicLocation),
  server: {
    port: 3000,
    middlewares: [
      staticFolder({
        root: Deno.cwd() + "/dist",
      }),
    ],
  },
}, {
  json: {
    extensions: {
      data: [".json"],
      pages: [".json"],
    },
  },
});

site.preprocess([".md", ".json", ".tmpl.ts"], (page) => {
    if (page.src.path.includes("comparePages")) {
      // here both data is included in `page` like expected
    }
});
...

Can you please point to the code where data is merged with the generator function?

[oscarotero]

If you want to dig into the code, this is the line where the date from the generator page and the yield object is merged:
https://github.com/lumeland/lume/blob/master/core/renderer.ts#L104

If the generator doesn't have this value, the new page neither.

The site/_data/config.yml is missing

_data files/folders inside the site/ folder are available only for pages in this folder and subfolders.
Maybe is this the problem?