Convert step-by-step guide to markdown #6326

forsyth2 · 2024-04-04T00:16:42Z

Converting the original step-by-step guide from Confluence to Markdown.

forsyth2 · 2024-04-04T00:17:14Z

Primary updates

Most obviously, the guide has been converted from Confluence to Markdown. Other differences:

The guide now refers to the v3 template script.
Cori references are replaced with Perlmutter.
I removed references to code older than Unified 1.6.0, since that is significantly out of date by now.
Similarly, I removed a warning about zstash check reports success even if exceptions occur zstash#167 since that has been resolved.
Misc. readability improvements

The full output can be seen at https://github.com/E3SM-Project/E3SM/blob/forsyth2/docs/update-guide/docs/step_by_step_guide.md

Documentation building

I followed the directions at Developing Documentation, but I ended up getting:

Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/mkdocs/__main__.py", line 192, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/mkdocs/commands/build.py", line 257, in build
    config = config['plugins'].run_event('config', config)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/mkdocs/plugins.py", line 102, in run_event
    result = method(item, **kwargs)
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/mkdocs_monorepo_plugin/plugin.py", line 52, in on_config
    new_docs_dir = self.merger.merge()
  File "/gpfs/fs1/home/ac.forsyth2/mkdocsenv/lib64/python3.6/site-packages/mkdocs_monorepo_plugin/merger.py", line 62, in merge
    copytree(source_dir, dest_dir, dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'

I even set up a new clone of E3SM to work in, and followed the relevant directions on the Development Getting Started Guide.

I wasn't able to build the docs before or after I made the changes here.

Other notes

I was trying to see if Confluence-to-Markdown exporters exist, but it looked like that could only be accomplished using non-Atlassian plugins. E.g., here and here. Markdown Exporter for Confluence is such an app.

But I ultimately decided to just port the guide over manually for two reasons:

There were a number of changes that needed to be made, so it gave me a chance to go through the guide line-by-line.
I wasn't sure about the security constraints of our Confluence space -- e.g., if we needed to get apps approved first. For example, Installing Marketplace apps mentions "A note about safety: apps are very powerful, they can change the behavior of almost any part of the Atlassian application. It's very important that you trust an app before you install it. Always be aware of where (and who) an app comes from."

github-actions · 2024-04-04T00:19:13Z

PR Preview Action v1.4.7
🚀 Deployed preview to https://E3SM-Project.github.io/E3SM/pr-preview/pr-6326/
on branch `gh-pages` at 2024-04-04 17:24 UTC

forsyth2

I've left some questions re: the best ways to update this guide.

docs/step_by_step_guide.md

forsyth2 · 2024-04-04T00:19:24Z

docs/step_by_step_guide.md

+# Documenting the Model Run
+You should create a Confluence page for your model run in the relevant Confluence space. Use the [Simulation Run Template](https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/2297299190) as a template. See below for how to fill out this template.
+
+<!-- TODO: where should the Confluence pages be made for v3 (and for each group)? -->


Is there a newer template that should be used? What Confluence directory should v3 simulations be documented in?

forsyth2 · 2024-04-04T00:20:04Z

docs/step_by_step_guide.md

+Make a bulleted list of links, e.g., for `<url_path>/ts_0001-0050_climo_0021-0050/`, create a bullet `“1-50 (time series), 21-50 (climatology)”.
+
+## ILAMB
+<!-- TODO: Add this section? -->


The original guide was written with Water Cycle in mind. Is there a preferred way to include ILAMB links in simulation documentation?

forsyth2 · 2024-04-04T00:20:42Z

docs/step_by_step_guide.md

+Update the simulation Confluence page with information regarding this simulation (For Water Cycle’s v2 work, that page is [V2 Simulation Planning](https://acme-climate.atlassian.net/wiki/spaces/ED/pages/2766340117)). In the `zstash archive` column, specify: 
+- `/home/<first letter>/<username>/E3SMv2/<case_name>`
+- `zstash_create_<stamp>.log`
+- `zstash_check_<stamp>.log`
+
+<!-- TODO: Is there a v3 planning page to link? -->


Is there a corresponding v3 simulation planning page with information about all the simulations?

forsyth2 · 2024-04-04T00:32:28Z

@mccoy20 and @chengzhuzhang requested a thorough update to the step-by-step guide, since it was written a while ago. @golaz @chengzhuzhang requested that the new version be done in Markdown rather than Confluence. I have implemented these requests here.

@golaz, @wlin7, @ndkeen -- @chengzhuzhang suggested you three as reviewers to check if the new guide is properly up-to-date. I'm therefore marking you all as reviewers. Thank you.

@rljacob, @wlin7 -- @chengzhuzhang suggested you two as being able to provide more instruction on including the Markdown guide into the E3SM Documentation. Thank you. I've followed the directions at Developing Documentation, but I wasn't able to get the documentation built (see note above). I'm also not quite sure how to link in the new guide. That page mentions a user-guide directory, but there is no such directory under E3SM/docs. Furthermore, it looks like most of the documentation covers the individual components (i.e., the index only links to components).

mahf708 · 2024-04-04T00:36:08Z

@forsyth2 the error is likely related to using python 3.6. Could you try with the latest e3sm-unified (1.9.3)? Just activate e3sm-unified and simply build with mkdocs build (no additional steps/packages are needed).

mahf708 · 2024-04-04T00:38:32Z

Also, could you please link this new file so that it appears in the docs? See preview of this PR here: https://e3sm-project.github.io/E3SM/pr-preview/pr-6326/. As an example of how to get this done, see the following change in PR #6246. (In other words, I would just link it on the main landing page as well as on the navigation panel, but others may have different preferences.)

forsyth2 · 2024-04-04T00:52:01Z

Could you try with the latest e3sm-unified (1.9.3)?

Ah, I didn't realize just using Unified would work. With that, I can start serving the documentation, but I now realize I have no way of looking at it, since it's serving locally on Chrysalis (where I do the bulk of my development) and not on my own machine. I figured just using the GitHub markdown rendering would work fine since it's just one page (as opposed to many connected ones).

could you please link this new file so that it appears in the docs?

The second commit includes this change.

mahf708 · 2024-04-04T00:54:45Z

mkdocs.yaml

@@ -3,8 +3,10 @@ site_name: E3SM
 nav:
  - Introduction: 'index.md'
  - 'Developing Docs': 'https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3924787306/Developing+Documentation'
+  - 'Step by Step Guide': 'https://e3sm-project.github.io/E3SM/pr-preview/pr-6326/'


Sorry, I may have misled you with my comment. This should be

Suggested change

- 'Step by Step Guide': 'https://e3sm-project.github.io/E3SM/pr-preview/pr-6326/'

- 'Step by Step Guide': 'step_by_step_guide.md'

@forsyth2 see a rendering of your additions here: https://e3sm-project.github.io/E3SM/pr-preview/pr-6326/step_by_step_guide/

mahf708 · 2024-04-04T01:02:02Z

but I now realize I have no way of looking at it, since it's serving locally on Chrysalis (where I do the bulk of my development) and not on my own machine. I figured just using the GitHub markdown rendering would work fine since it's just one page (as opposed to many connected ones).

We have the preview in the PR to help users: #6326 (comment).

However, just in case it is helpful, you could do the following

tunnel a port to chrysalis from your local machine (e.g., ssh -L 1234:localhost:1234 [email protected])
mkdocs serve with that port (e.g., mkdocs serve -a localhost:1234)
then, open the link it prints in your browser (e.g., localhost:1234)

forsyth2

Self-review using the actual rendering and the GitHub rendering.

There are a large number of things that render fine in the GitHub rendering, but not the actual rendering. Are we actually using a different variant of Markdown? Or is the document rendering software we use only capable of handling a subset of Markdown? In any case, the line "Our documentation is written in Markdown language. See Basic writing and formatting syntax - GitHub Docs to get started." in Developing Documentation should be updated to reflect the fact that a great number of useful features don't actually work as in GitHub...

docs/step_by_step_guide.md

forsyth2 · 2024-04-04T16:17:42Z

docs/step_by_step_guide.md

+- Chrysalis – `source ~/.bashrc`
+- Compy – `source ~/.bash_profile`
+- Perlmutter – `source ~/.bash_profile.ext`


These bullets show up fine on the GitHub rendering but all in one line on the actual rendering...

This problem recurs throughout the file.

This and others can be fixed. Do you want me to go through it and help guide edits to meet the mkdocs rendering?

Yes that would be great. Thanks @mahf708! (I still think the docs on developing our docs should be updated to reflect that we have some differences from standard GitHub markdown.)

docs/step_by_step_guide.md

forsyth2 · 2024-04-04T16:20:08Z

docs/step_by_step_guide.md

+<run_scripts_dir>: `${HOME}/E3SM/scripts`
+
+<code_source_dir>: `${HOME}/E3SM/code`
+
+<simulations_dir> differs amongst machines:


These show up fine in the GitHub rendering, but in the actual rendering, anything in <> brackets simply shows up as an empty string...

This problem recurs throughout the file.

forsyth2 · 2024-04-04T16:23:13Z

docs/step_by_step_guide.md

+# Configuring the Model Run – Run Script
+Start with an example of a run script for a low-resolution coupled simulation:  
+
+We'll use the template script in the E3SM repository, which uses Perlmutter: https://github.com/E3SM-Project/E3SM/blob/master/run_e3sm.template.sh


This link works in the GitHub rendering, but not the actual rendering. In any case, I'm replacing it with this line:
We'll use the [template script](https://github.com/E3SM-Project/E3SM/blob/master/run_e3sm.template.sh) in the E3SM repository, which uses Perlmutter.

forsyth2 · 2024-04-04T16:27:04Z

docs/step_by_step_guide.md

+> [!IMPORTANT]
+> If this is part of a simulation campaign, ask your group lead about using a `CASE_GROUP` label. Otherwise, please use a unique name to distinguish from existing `CASE_GROUP` label names, (e.g., “v2.LR“).


This note box works in the GitHub rendering, but not fully correct in the actual rendering. It does show up as a sidebar/quote block, but the information about the note type is lost (and it just prints the [!IMPORTANT] as if it were regular text).

This problem recurs throughout the file.

forsyth2 · 2024-04-04T16:29:22Z

docs/step_by_step_guide.md

+1. The model starts without errors.
+2. The model produces BFB (bit-for-bit) results after a restart.
+3. The model produces BFB results when changing PE layout.


The numbered list works in the GitHub rendering, but not in the actual rendering (where it shows up all in one line).

forsyth2 · 2024-04-04T16:33:39Z

docs/step_by_step_guide.md

+| Component | Directory | File naming pattern |
+| --- | --- | --- |
+| Atmosphere (Earth Atmospheric Model) | `archive/atm/hist` | `*.eam.h*` |
+| Coupler | `archive/cpl/hist` | `*.cpl.h*` |
+| Sea Ice (MPAS-Sea-Ice) | `archive/ice/hist` | `*.mpassi.hist.*` |
+| Land (Earth Land Model) | `archive/lnd/hist` | `*.elm.h*` |
+| Ocean (MPAS-Ocean) | `archive/ocn/hist` | `*.mpaso.hist.*` |
+| River Runoff (MOSART) | `archive/rof/hist` | `*.mosart.h*` |


The table shows up in the GitHub rendering, but all as one line in the actual rendering...

forsyth2 · 2024-04-04T16:34:17Z

docs/step_by_step_guide.md

+Use `--force-move` to move instead of copying, which can take a long time. Set `--last-date` to the latest date in the simulation you want to archive. You do not have to specify a beginning date.
+```shell
+cd <simulations_dir>/<case_name>/case_scripts
+./case.st_archive --last-date 0051-01-01 --force-move --no-incomplete-logs


0051 shows up as red in the actual rendering for some reason

rljacob · 2024-04-04T16:48:12Z

@forsyth2 regarding rendering diffs: the docs are converted to html from markdown by mkdocs and the html is what gets displayed on the github.io page.

forsyth2 · 2024-04-04T16:50:29Z

regarding rendering diffs: the docs are converted to html from markdown by mkdocs and the html is what gets displayed on the github.io page.

I'm still a little confused here. Does that mean mkdocs converts markdown -> html differently than how the GitHub website does it for things like READMEs and these pull request comments?

rljacob · 2024-04-04T17:24:44Z

Yes its a different rendering engine. mkdocs has options to further change the rendered look and feel. That's why the our documentation docs tell the user to use the mkdocs built-in html server to look at the result.

rljacob · 2024-04-11T16:36:48Z

If you want, this could be added at a higher level in the github docs: https://e3sm-project.github.io/. To do that, make a PR to https://github.com/E3SM-Project/E3SM-Project.github.io

forsyth2 · 2024-04-11T16:54:46Z

@rljacob That sounds good to me. I'll do that, thanks!

forsyth2 · 2024-04-11T17:28:05Z

Closing in favor of E3SM-Project/E3SM-Project.github.io#2

Convert step-by-step guide to markdown

2317a61

forsyth2 added the Documentation label Apr 4, 2024

forsyth2 self-assigned this Apr 4, 2024

forsyth2 commented Apr 4, 2024

View reviewed changes

forsyth2 mentioned this pull request Apr 4, 2024

Convert step-by-step guide to markdown forsyth2/E3SM#1

Closed

Address comments

5d18f8c

mahf708 reviewed Apr 4, 2024

View reviewed changes

forsyth2 requested review from ndkeen, wlin7 and golaz April 4, 2024 01:13

Address comments 2

3d76bc1

forsyth2 commented Apr 4, 2024

View reviewed changes

forsyth2 force-pushed the forsyth2/docs/update-guide branch from c1bfcc8 to 56ba2cc Compare April 4, 2024 16:42

Address self-review comments

6e9ab8f

forsyth2 force-pushed the forsyth2/docs/update-guide branch from 56ba2cc to 6e9ab8f Compare April 4, 2024 16:44

forsyth2 mentioned this pull request Apr 11, 2024

Convert step-by-step guide to markdown E3SM-Project/E3SM-Project.github.io#2

Merged

forsyth2 closed this Apr 11, 2024

forsyth2 deleted the forsyth2/docs/update-guide branch April 11, 2024 17:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert step-by-step guide to markdown #6326

Convert step-by-step guide to markdown #6326

forsyth2 commented Apr 4, 2024

forsyth2 commented Apr 4, 2024 •

edited

Loading

github-actions bot commented Apr 4, 2024 •

edited

Loading

forsyth2 left a comment

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 commented Apr 4, 2024

mahf708 commented Apr 4, 2024 •

edited

Loading

mahf708 commented Apr 4, 2024 •

edited

Loading

forsyth2 commented Apr 4, 2024

mahf708 Apr 4, 2024 •

edited

Loading

mahf708 Apr 4, 2024

mahf708 commented Apr 4, 2024 •

edited

Loading

forsyth2 left a comment

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

mahf708 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

forsyth2 Apr 4, 2024

rljacob commented Apr 4, 2024

forsyth2 commented Apr 4, 2024

rljacob commented Apr 4, 2024

rljacob commented Apr 11, 2024 •

edited

Loading

forsyth2 commented Apr 11, 2024

forsyth2 commented Apr 11, 2024

	- 'Step by Step Guide': 'https://e3sm-project.github.io/E3SM/pr-preview/pr-6326/'
	- 'Step by Step Guide': 'step_by_step_guide.md'

		> [!IMPORTANT]
		> If this is part of a simulation campaign, ask your group lead about using a `CASE_GROUP` label. Otherwise, please use a unique name to distinguish from existing `CASE_GROUP` label names, (e.g., “v2.LR“).

Convert step-by-step guide to markdown #6326

Convert step-by-step guide to markdown #6326

Conversation

forsyth2 commented Apr 4, 2024

forsyth2 commented Apr 4, 2024 • edited Loading

Primary updates

Documentation building

Other notes

github-actions bot commented Apr 4, 2024 • edited Loading

forsyth2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

forsyth2 commented Apr 4, 2024

mahf708 commented Apr 4, 2024 • edited Loading

mahf708 commented Apr 4, 2024 • edited Loading

forsyth2 commented Apr 4, 2024

mahf708 Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahf708 commented Apr 4, 2024 • edited Loading

forsyth2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rljacob commented Apr 4, 2024

forsyth2 commented Apr 4, 2024

rljacob commented Apr 4, 2024

rljacob commented Apr 11, 2024 • edited Loading

forsyth2 commented Apr 11, 2024

forsyth2 commented Apr 11, 2024

forsyth2 commented Apr 4, 2024 •

edited

Loading

github-actions bot commented Apr 4, 2024 •

edited

Loading

mahf708 commented Apr 4, 2024 •

edited

Loading

mahf708 commented Apr 4, 2024 •

edited

Loading

mahf708 Apr 4, 2024 •

edited

Loading

mahf708 commented Apr 4, 2024 •

edited

Loading

rljacob commented Apr 11, 2024 •

edited

Loading