Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Follow-up / Making Scipio environment-configurable #154

Merged
merged 10 commits into from
Nov 18, 2024
17 changes: 13 additions & 4 deletions Sources/ScipioKit/DescriptionPackage.swift
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,21 @@ struct DescriptionPackage: PackageLocator {
/// Then, use package versions only from existing Package.resolved.
/// If it is `true`, Package.resolved never be updated.
/// Instead, the resolving will fail if the Package.resolved is mis-matched with the workspace.
init(packageDirectory: ScipioAbsolutePath, mode: Runner.Mode, onlyUseVersionsFromResolvedFile: Bool) throws {
init(
packageDirectory: ScipioAbsolutePath,
mode: Runner.Mode,
onlyUseVersionsFromResolvedFile: Bool,
toolchainEnvironment: ToolchainEnvironment? = nil
) throws {
self.packageDirectory = packageDirectory
self.mode = mode

let toolchain = try UserToolchain(swiftSDK: try .hostSwiftSDK())
self.toolchain = toolchain
self.toolchain = try UserToolchain(
swiftSDK: try .hostSwiftSDK(
toolchainEnvironment?.toolchainBinPath,
environment: toolchainEnvironment.asSwiftPMEnvironment
),
environment: toolchainEnvironment.asSwiftPMEnvironment
)
Copy link
Contributor Author

@dmhts dmhts Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the belated response. I'm back as of today!

We’re almost finished with our Scipio-powered tool, and during final testing, we discovered another environment-related issue (hopefully the last one!).

There’s a bug in SwiftPM where Environment isn’t passed down to the shell-executing function here. Fortunately, it’s possible to work around this by supplying a custom binDir (pointing to the bin dir of the current toolchain in our case) a few lines above.

I’m planning to file another fix for this in SwiftPM soon (I've already merged these two, and have one currently open). Although it’s not a critical issue, resolving it would allow a cleaner setup by removing the need to specify a custom bin path.

Apologies for including this change in the current PR, but I believe it’s closely related to its focus, and it’s purely additive, leaving existing behavior unaffected.

As mentioned, this appears to be the last environment-related enhancement needed to enable Scipio for multi-toolchain use. 🎉


let workspace = try Self.makeWorkspace(toolchain: toolchain, packagePath: packageDirectory)
let scope = makeObservabilitySystem().topScope
Expand Down
7 changes: 4 additions & 3 deletions Sources/ScipioKit/Producer/FrameworkProducer.swift
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ struct FrameworkProducer {
private let overwrite: Bool
private let outputDir: URL
private let fileSystem: any FileSystem
private let toolchainEnvironment: [String: String]?
private let toolchainEnvironment: ToolchainEnvironment?

private var cacheStorage: (any CacheStorage)? {
switch cacheMode {
Expand Down Expand Up @@ -52,7 +52,7 @@ struct FrameworkProducer {
cacheMode: Runner.Options.CacheMode,
overwrite: Bool,
outputDir: URL,
toolchainEnvironment: [String: String]? = nil,
toolchainEnvironment: ToolchainEnvironment? = nil,
fileSystem: any FileSystem = localFileSystem
) {
self.descriptionPackage = descriptionPackage
Expand Down Expand Up @@ -280,7 +280,8 @@ struct FrameworkProducer {
let compiler = PIFCompiler(
descriptionPackage: descriptionPackage,
buildOptions: buildOptions,
buildOptionsMatrix: buildOptionsMatrix
buildOptionsMatrix: buildOptionsMatrix,
toolchainEnvironment: toolchainEnvironment
)
try await compiler.createXCFramework(buildProduct: product,
outputDirectory: outputDir,
Expand Down
4 changes: 2 additions & 2 deletions Sources/ScipioKit/Producer/PIF/PIFCompiler.swift
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ struct PIFCompiler: Compiler {
private let fileSystem: any FileSystem
private let executor: any Executor
private let buildOptionsMatrix: [String: BuildOptions]
private let toolchainEnvironment: [String: String]?
private let toolchainEnvironment: ToolchainEnvironment?

private let buildParametersGenerator: BuildParametersGenerator

init(
descriptionPackage: DescriptionPackage,
buildOptions: BuildOptions,
buildOptionsMatrix: [String: BuildOptions],
toolchainEnvironment: [String: String]? = nil,
toolchainEnvironment: ToolchainEnvironment? = nil,
fileSystem: any FileSystem = TSCBasic.localFileSystem,
executor: any Executor = ProcessExecutor()
) {
Expand Down
47 changes: 41 additions & 6 deletions Sources/ScipioKit/Producer/PIF/ToolchainGenerator.swift
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ struct ToolchainGenerator {
let destination: SwiftSDK = try await makeDestination(sdk: sdk)
return try UserToolchain(
swiftSDK: destination,
environment: environment.map(Environment.init) ?? .current
environment: environment.asSwiftPMEnvironment
)
}

Expand All @@ -44,11 +44,11 @@ struct ToolchainGenerator {
// Compute common arguments for clang and swift.
var extraCCFlags: [String] = []
var extraSwiftCFlags: [String] = []
let sdkPaths = try SwiftSDK.sdkPlatformFrameworkPaths(environment: [:])
extraCCFlags += ["-F", sdkPaths.fwk.pathString]
extraSwiftCFlags += ["-F", sdkPaths.fwk.pathString]
extraSwiftCFlags += ["-I", sdkPaths.lib.pathString]
extraSwiftCFlags += ["-L", sdkPaths.lib.pathString]
let macosSDKPlatformPaths = try await resolveSDKPlatformFrameworkPaths()
extraCCFlags += ["-F", macosSDKPlatformPaths.frameworkPath.pathString]
extraSwiftCFlags += ["-F", macosSDKPlatformPaths.frameworkPath.pathString]
extraSwiftCFlags += ["-I", macosSDKPlatformPaths.frameworkPath.pathString]
extraSwiftCFlags += ["-L", macosSDKPlatformPaths.frameworkPath.pathString]
dmhts marked this conversation as resolved.
Show resolved Hide resolved

let buildFlags = BuildFlags(cCompilerFlags: extraCCFlags, swiftCompilerFlags: extraSwiftCFlags)
return SwiftSDK(
Expand All @@ -60,3 +60,38 @@ struct ToolchainGenerator {
)
}
}

extension ToolchainGenerator {

/// A non-caching environment-aware implementation of `SwiftSDK.sdkPlatformFrameworkPaths`
/// This implementation is based on the original SwiftPM
/// https://github.com/swiftlang/swift-package-manager/blob/release/6.0/Sources/PackageModel/SwiftSDKs/SwiftSDK.swift#L592-L595
/// Returns `macosx` sdk platform framework path.
fileprivate func resolveSDKPlatformFrameworkPaths() async throws -> (frameworkPath: AbsolutePath, libPath: AbsolutePath) {
let platformPath = try await executor.execute(
"/usr/bin/xcrun",
"--sdk",
"macosx",
"--show-sdk-platform-path"
)
.unwrapOutput()
.spm_chomp()

guard !platformPath.isEmpty else {
throw StringError("could not determine SDK platform path")
}

// For XCTest framework.
let frameworkPath = try AbsolutePath(validating: platformPath).appending(
components: "Developer", "Library", "Frameworks"
)

// For XCTest Swift library.
let libPath = try AbsolutePath(validating: platformPath).appending(
components: "Developer", "usr", "lib"
)

return (frameworkPath, libPath)
}

}
10 changes: 6 additions & 4 deletions Sources/ScipioKit/Runner.swift
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@ public struct Runner {
descriptionPackage = try DescriptionPackage(
packageDirectory: packagePath,
mode: mode,
onlyUseVersionsFromResolvedFile: false
onlyUseVersionsFromResolvedFile: false,
toolchainEnvironment: options.toolchainEnvironment
)
} catch {
throw Error.invalidPackage(packageDirectory)
Expand All @@ -99,7 +100,8 @@ public struct Runner {
buildOptionsMatrix: buildOptionsMatrix,
cacheMode: options.cacheMode,
overwrite: options.overwrite,
outputDir: outputDir
outputDir: outputDir,
toolchainEnvironment: options.toolchainEnvironment
)
do {
try await producer.produce()
Expand Down Expand Up @@ -240,7 +242,7 @@ extension Runner {
public var cacheMode: CacheMode
public var overwrite: Bool
public var verbose: Bool
public var toolchainEnvironment: [String: String]?
public var toolchainEnvironment: ToolchainEnvironment?

public init(
baseBuildOptions: BuildOptions = .init(),
Expand All @@ -249,7 +251,7 @@ extension Runner {
cacheMode: CacheMode = .project,
overwrite: Bool = false,
verbose: Bool = false,
toolchainEnvironment: [String: String]? = nil
toolchainEnvironment: ToolchainEnvironment? = nil
) {
self.buildOptionsContainer = BuildOptionsContainer(
baseBuildOptions: baseBuildOptions,
Expand Down
19 changes: 19 additions & 0 deletions Sources/ScipioKit/ToolchainEnvironment.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import struct Basics.AbsolutePath
@_spi(SwiftPMInternal) import struct Basics.Environment

public typealias ToolchainEnvironment = [String: String]
Copy link
Contributor Author

@dmhts dmhts Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giginet I've extracted all the ToolchainEnvironment-related logic into a separate file and introduced a type alias for improved semantics. Does that work for you?


extension ToolchainEnvironment {
var developerDirPath: String? { self["DEVELOPER_DIR"] }
var toolchainBinPath: Basics.AbsolutePath? {
if let developerDirPath {
return try? AbsolutePath(validating: developerDirPath)
.appending(components: "Toolchains", "XcodeDefault.xctoolchain", "usr", "bin")
}
return nil
}
}

extension ToolchainEnvironment? {
Copy link
Owner

@giginet giginet Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's confusing to extend to Optional.
.current is injected on the callee side, but I prefer to inject on the caller side explicitly.

Copy link
Contributor Author

@dmhts dmhts Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is to shift all the boilerplate from the calling site to the extension. Sure, we can use map or flatMap at the calling site, but then it would be like this:

try UserToolchain(
    swiftSDK: try .hostSwiftSDK(
        toolchainEnvironment?.toolchainBinPath,
        environment: toolchainEnvironment.flatMap { Environment($0) } ?? Environment.current
    ),
    environment: toolchainEnvironment.flatMap { Environment($0) } ?? Environment.current
)

versus with the optional extension:

try UserToolchain(
    swiftSDK: try .hostSwiftSDK(
        toolchainEnvironment?.toolchainBinPath,
        environment: toolchainEnvironment.asSwiftPMEnvironment
    ),
    environment: toolchainEnvironment.asSwiftPMEnvironment
)

Additionally, in the first case, we would also need to use this clunky import: @_spi(SwiftPMInternal) import struct Basics.Environment every time Environment is referenced.

or did you mean something else?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the comment. I haven't considered _spi.

I already tweaked my changes, but I'll revert this commit. 260baa5

Sorry for the modification!

Copy link
Contributor Author

@dmhts dmhts Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@giginet I see what you mean! Looks good to me 👍

Update: yeah, it doesn't solve the issue with the import.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I already reverted the commit 😓

If you think it's good, I'll apply the commit again and merge this. Thank you!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was mistaken, the import was not needed in your case. Yeah, feel free to apply it again. As a benefit of your approach - it's more obvious that the variable is optional at the calling site.

var asSwiftPMEnvironment: Environment { map(Environment.init) ?? Environment.current }
}
19 changes: 19 additions & 0 deletions Tests/ScipioKitTests/ToolchainEnvironmentTests.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import XCTest
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use @Testing for new tests 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't noticed you already used it in the other tests. No problem, I'll do it next time 👍

@testable import ScipioKit
@_spi(SwiftPMInternal) import struct Basics.Environment

final class ToolchainEnvironmentTests: XCTestCase {

func testBasics() {
let environment = [
"DEVELOPER_DIR": "/Applications/Xcode.app/Contents/Developer",
]

XCTAssertEqual(environment.developerDirPath, "/Applications/Xcode.app/Contents/Developer")
XCTAssertEqual(
environment.toolchainBinPath?.pathString,
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin"
)
}

}