Revisit support for Spack target architectures, for best fit with Wave requirements

[marcodelapierre]

We want to make sure we are able to build Wave Containers with target micro-architectures that are relevant/useful to the service.

Current status:

• Nextflow is capable of feeding fine grained CPU archs, through spackArch when using the target subdirective, see L65 in https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/test/groovy/nextflow/processor/ArchitectureTest.groovy#L63-L65
• The Wave Client in the Nextflow Wave Plugin indeed uses spackArch, see

  nextflow/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy

  Line 385 in ec156bc

  final spackArch = arch ? arch.spackArch : DEFAULT_SPACK_ARCH

• The current default in the Wave Client is currently x86_64, see

  nextflow/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy

  Line 85 in ec156bc

  private static final String DEFAULT_SPACK_ARCH = 'x86_64'

• [UPDATED] The client hard codes submitting the dockerArch as the platform to Wave, therefore losing any information of the Spack target, see

  nextflow/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy

  Lines 490 to 506 in ec156bc

  	final platform = dockerArch
  	// check is a valid container image
  	WaveAssets.validateContainerName(containerImage)
  	// read the container config and go ahead
  	final containerConfig = this.resolveContainerConfig(platform)
  	return new WaveAssets(
  	containerImage,
  	platform,
  	bundle,
  	containerConfig,
  	containerScript,
  	condaFile,
  	spackFile,
  	projectRes,
  	singularity)
  	}

TO-DO:

• Nextflow side: change default to x86_64_v3
• [UPDATED] Nextflow side: diversify between container platform required for container builder and target for Spack commands in containerfile (coordinated with Wave backend)
• Wave backend side: see Revisit support for Spack target architectures, to match Nextflow inputs seqeralabs/wave#336

For reference, list of Spack supported targets as of today: https://spack.readthedocs.io/en/latest/basic_usage.html#support-for-specific-microarchitectures

[pditommaso]

What's the difference of x86_64 vs x86_64_v3?

[marcodelapierre]

x86_64_v3 is a CPU family that corresponds to newer CPUs; it features a number of additional features, such as AVX2 (256 bit floating point registers for vector arithmetics). Compiling for it results in more performant binaries than for x86_64. Pretty much all CPUs marketed after 2015 fall in this family, so there is very, very little risk of incompatibility.

[pditommaso]

OK, but still there's the need to support the target microarchitecture then

[marcodelapierre]

Yes, in Spack jargon: name: 'linux/x86_64', target: 'x86_64_v3'

I have updated the original post above - indeed at first I had missed the point you have just made

[pditommaso]

I see. Main problem there's no support for target arch in the current request object

nextflow/plugins/nf-wave/src/main/io/seqera/wave/plugin/SubmitContainerTokenRequest.groovy

Lines 34 to 122 in d2dd216

	/**
	* Tower access token required to enable the service
	*/
	String towerAccessToken
	/**
	* Tower refresh token
	*/
	String towerRefreshToken
	/**
	* Tower workspace id
	*/
	Long towerWorkspaceId
	/**
	* Tower endpoint
	*/
	String towerEndpoint
	/**
	* The ID of the workflow that submitted this container request
	*/
	String workflowId
	/**
	* Container image to be pulled
	*/
	String containerImage
	/**
	* Container build file i.g. Dockerfile of the container to be build
	*/
	String containerFile
	/**
	* List of layers to be added in the pulled image
	*/
	ContainerConfig containerConfig
	/**
	* Conda recipe file used to build the container
	*/
	String condaFile
	/**
	* Spack recipe file used to build the container
	*/
	String spackFile
	/**
	* The request container platform
	*/
	String containerPlatform
	/**
	* The target repository where the built container needs to be stored
	*/
	String buildRepository
	/**
	* The container repository to cache build layers
	*/
	String cacheRepository
	/**
	* Request
	*/
	String timestamp
	/**
	* Request unique fingerprint
	*/
	String fingerprint
	/**
	* Enable freeze container mode
	*/
	boolean freeze
	/**
	* Specify the format of the container file
	*/
	String format
	/**
	* When {@code true} build requests are carried out in dry-run mode.
	*/
	Boolean dryRun

[marcodelapierre]

exactly, we will need to add it