2020 Albany Users Meeting (Jan 27 28, Albuquerque)

The 5th Albany User Meeting (AUM) will be held on Monday-Tuesday January 27-28, 2020 at the Computer Science Research Institute (CSRI) building (room 279) at the Sandia National Laboratories site in Albuquerque, NM.

The purpose of AUM is to bring together Albany users and developers to exchange ideas, discuss developments, and to foster the growth of Albany as a powerful tool for computational methods and applications in mechanics, engineering and applied sciences. The topics covered will include:

• The current state of Albany
• General Albany development, issues and concerns.
• Developments among different projects (LandIce, LCM, AMP, etc.).
• Future directions.

Logistics

The CSRI building is located at 1450 Innovation Pkwy SE, Albuquerque, 87123. In order to gain access to CSRI, you need to get your badge at the badge office in the IPOC building, located at 1611 Innovation Pkwy Albuquerque, 87123. That's just half a mile from the CSRI building. Both the buildings are outside the Kirtland Air Force Base (KAFB). Keep in mind that Mondays can be a little busier at the badge office, so plan accordingly. We start at 9am, so I would suggest to be at the badge office no later than 8:30am, to avoid surprises. Bring your photo ID (must be a real ID, so make sure your state issued ID is compliant with federal regulations). If your a foreign national, bring your passport, with your visa/EAD card. If you're a permanent resident, bring your LPR card (green card). If you are a naturalized US citizen, you may need naturalization documents in addition to your ID (we had a person denied a badge cause they only brought their US passport, but the system showed him as a naturalized citizen, and badge office personnel required naturalization docs, which this person didn't have).

For lunch, we usually have a different food truck every day down the street (though Monday is not the greatest). Jimmy John's is also very close, and we can have sandwiches/coffee ordered in. We will have a few options for lunch/dinner to vote on first thing Monday morning.

Presentations (Monday, 9am -5pm MST)

Presentations should be no longer than 30min, including time for discussion/questions. Therefore, you should shoot for a 20-25m presentation. Here's a tentative list of presenters, with their title/topic of the presentation.

0. (9:00- 9:30 MST) Andy Salinger, Luca Bertagna: Welcome remarks, logistic info
1. (9:30-10:00 MST) Josh Robbins: "Albany and Plato Engine: Multi-objective optimization using MPMD" [PDF] [PPT]
2. (10:00-10:30 MST) Tim Fuller: "PFLOTRAN: Implementing an external flow code as an Albany evaluator" [PDF]
3. (10:30-11:00 MST) Irina Tezaur: "The Schwarz alternating method for multiscale coupling in solid mechanics" [PPT] [PDF]
4. (11:00-12:00 MST) Roger Pawlowski [Skype talk]: "Hierarchic Parallelism, Memory Management and Helpful Debugging Tools: Updates on Trilinos Discretization Packages" [PDF]
5. (12:00-1:30 MST) Lunch
6. (1:30-2:00 MST) Alejandro Mota: "Strongly-coupled thermo-mechanical model of permafrost for the simulation of Arctic coastal erosion" [PDF] [PPT]
7. (2:00-2:30 MST) Mauro Perego: "Albany Land Ice (ALI) model, focus on optimization capability" [PDF]
8. (2:30-3:00 MST) Jerry Watkins: "Performance portability in Albany" [PDF]
9. (3:00- 3:30 MST) Luca Bertagna: "2D-3D coupling struggles: subglacial hydrology in Albany-LandIce" [PDF]
10. (3:30- 4:00 MST) Ruixiong Hu: "Finite Element Modeling of Additive and Subtractive Manufacturing accounting for Geometry Evolution" [PDF]
11. (4:00- 4:30 MST) Jacob Pessin: "A Thermal-Mechanical Finite Element Formulation for Simulating Additive and Subtractive Manufacturing" [PDF]
12. (4:30- 5:00 MST) Justin Clough: "Inelastic deformation of a panel subject to hypersonic loads" [PDF]

Discussions (Tuesday, 9am-1pm MST)

Tuesday will be focused on discussions on development/planning for Albany. You are welcome to edit this list to add your topic of discussion. We are trying to keep the end time around 1pm, so that people coming in from out of town can catch an afternoon flight if they want. We don't have a specific time limit per topic (some may be faster than others), but given the 4h total time, we should aim at 1h per topic.

• Block discretization: this is a major topic, which we've been thinking about for over a year. It would allow Albany to tackle more complex problems (e.g., mixed FE discretizations, 2D-3D couplings) without the need of ad-hoc tricks, and deep code hacks. The discussion could/should cover timings, development strategies, legacy support, and all the ramifications that such effort could entail.
• Performance maintenance/monitoring: since Albany is used in production runs at large scales, it is essential to keep our performance efforts on track. Topics can include:
  • Portability (e.g., CPU/GPU support)
  • Regression testing
  • Ideas for handling current bottlenecks
  • Tools for performance testing
• Maintaining software quality: how can we prevent Albany from breaking?
  • Continuous integration (CI) testing
  • Integrating better with Trilinos team - how can we make it so Trilinos does not break Albany as often (e.g. push Trilinos folks to test with same environments / on same machines as we do)
  • 2-branch workflow?
  • Move CDash site to one supported by SEMS (perhaps SEMS folks can help us in other ways with code maintenance?)
  • To fork or not to fork, that is the question? How to deal with forked versions of Albany going forward.
• Usability: how can we make Albany easier to use/develop?
  • Reviving/maintaining SPACK build originally created by @gahansen
  • How to get people to develop in master rather than in branches/forks?
  • Ownership of Albany
  • Documentation/outreach?

Discussion Notes

• Usability:

  • Documentation would be great!
  • Would be great to have documentation for input files, including solver settings.
  • RPI has some documentation, currently on Google drive - can share it with people so they can add to it.
  • Have Albany --help?
  • There is some documentation for adding new physics here. Irina has some slides about Albany which can be shared (after being brought to be up-to-date).
  • Eclipe can help figure out how code is structures (Jerry).
  • Need to clean up doc folder and Demo PDEs.
    • Cahn Hilliard problem? Glen wrote, has to do with phase field model.
    • utilities folder is kind of clogged.
  • Separate user and developer manual?
    • doxygen? Luca is against - people do not put much in doxygen, just tiny decoration...
  • Action items:
    • Irina: get R&A for slides on Albany development, put into latex file
    • Everyone: clean up/add to latex file
    • Do a survey of what documentation we do have:
      • RPI folks: share documentation so we can have a look at it.
      • Irina: find Albany user manual that was started awhile ago by LCM guys and survey
    • Have a description in header file of what problem you are solving, what are units (and/or add reference to book/paper, with page number). See this evaluator for an example of documenting PDE, and this for documenting units.
  • Reduce defaults! We have been burned a lot of times with this!
  • Reduce ifdef guards, e.g. LCM in this evaluator.
    • If you are putting a lot of ifdefs in general code, consider creating another evaluator that inherits from the main one.
  • SPACK build?
    • Tim Fuller did a lot of work with SPACK and Alegra. Alegra uses its own branch of SPACK, which will never go into master SPACK.
    • We can have a SPACK fork as a submodule in Albany.
    • Glen: SPACK can be really great for users who have never compiled the code.
    • Irina TODO: move Albany fork to SNLComputation github repo, scope out creating nightly build with SPACK.

• Block discretization:

  • See here for old notes on how to go about this.
  • Block gather/scatter
    • Panzer has block gather/scatter (Roger)
      • Panzer originally had 32 gather/scatter combinations (block/non-block, Tpetra/Epetra, etc.) - trying to drop everything except blocks
      • Teko preconditioning was a complication - all preconditioners need to be wrapped in Teko, which makes input file more complicated
      • Block DOFManager, block gather/scatter, Teko for preconditioning - 3 main places to explore (where everything comes in together for blocks)
      • Look at Panzer_ScatterResidual_BlockedTpetra_impl.cpp
      • Hijack Panzer block unit tester to unit test blocked gather/scatter
  • Output
    • If have same mesh for 2 physics, need to be careful that write different fields correctly to mesh (i.e., that writeSolution gets called with right index, etc.).
  • Multiple FieldManagers for 2D and 3D equations? Similar to volume and sideset FieldManagers.
  • Blocked DOFManager in Panzer - will be deprecated at some point (wraps vector of DOFManagers).
    • just work with vector of DOFManagers.
    • DOFManager supports multiple element blocks, as long as they are same cell topology

• Maintenance:

  • How to deal with forks, e.g. LCM fork for ACE created recently by @lxmota?
    • Unclear what are long term intentions for this fork...
  • Create new directory with unit tests - gtest or catch libraries for unit tests (macro-based, header only)
    • Glen prefers gtest; Jerry also uses gtest
    • Good for block discretization as we start to develop it
  • Continuous integration (CI) testing
    • Travis CI has limitations, e.g. 1 hour time limit, which are not feasible
    • Luca proposes using Autotester, developed by SEMS group, and works on SNL machines
      • Master Jenkins job launches separate Jenkins jobs either on SRN or SON machines - Luca does not have access to SRN Jenkins job.
      • Autotester is internal to Sandia
      • Regardless, preventing merge to master is probably a good idea. In this case, set of admins will have permission to push to master. Everyone else will have to submit PRs.
        • To discuss at Albany weekly meeting next week. Also to discuss: who will have permissions to push to master? Encourage PRs at next Albany meeting.
      • How much time will it take to maintain Autotester builds? Is it worth developers time to maintain this?

• Performance testing:

  • Luca: on CPUs we should run each test multiple times (e.g. 5) to take into account variability that you see on CPUs.
    • Jerry: some of the tests are too slow for multiple runs...
    • Request: add a plot with a window (moving) average on Jupyter notebook plots (e.g. here.
  • ProSPect-specific question: should SNL take over setting up MALI nightlies? ORNL was supposed to do this...
    • Irina: we will want to get Joe Kennedy's scripts for pushing to CDash site (nontrivial b/c build/test system is not cmake/ctest).
    • There are different branches, which might complicate things...
    • Purpose would be correctness, not performance
  • ML vs. MueLu performance for ProSPect
    • TODO: make ML and MueLu performance tests have comparable settings (work with Ray).
    • TODO: work with Ray to create good MueLu settings for MPAS, and switch to MueLu/Tpetra to give to client scientist collaborators, and to add to nightly testing.
  • KNL vs. Haswell on Cori - KNL will be slower on Haswell even with OpenMP... but they are charging less per CPU/hour to run on KNL, and getting through the queue will be faster.
    • KNL will be maybe 20% slower - may be worth it still given the situation above.
    • TODO (Jerry): create OpenMP KNL build on Cori, and provide input files.
    • Larger problems (e.g. Antarctica) may not fit on Cori KNL, unless a ton of nodes are used.
    • KNL nodes have less memory relative to MPI ranks

Skype for Business/Call-in Information

The Albany User Meeting presentations and discussion will be live-streamed using Skype for Business. Here is the Skype information to join the meeting:

Join Skype Meeting<https://meeting.sandia.gov/dsflane/06DZQ47P>
Trouble Joining? Try Skype Web App<https://meeting.sandia.gov/dsflane/06DZQ47P?sl=1>

Join by phone: 1-505-844-5300, Conference ID: 479716239

If you are outside of Sandia and your institution does not have Skype for Business, the only option to join is by phone. In this case, you will be able to hear the meeting, but not see the slides/streamed content, regrettably.