Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EPIC: Modify Docker build to create Deterministic Images #332

Open
3 tasks
Tracked by #366
AlfredoG87 opened this issue Nov 8, 2024 · 14 comments
Open
3 tasks
Tracked by #366

EPIC: Modify Docker build to create Deterministic Images #332

AlfredoG87 opened this issue Nov 8, 2024 · 14 comments
Assignees
Labels
Block Node Issues/PR related to the Block Node. Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. Release Issues tied to release management functions

Comments

@AlfredoG87
Copy link
Contributor

AlfredoG87 commented Nov 8, 2024

Problem

As a consumer of block node application docker images
I want the images to be deterministic
So that, I could reproduce the same image to verify its contents

We need the builds from the block-node docker images to be deterministic

Solution

  • Integrate the gradle plugin as it is on hedera-services so that we get deterministic software builds
  • Use the hedera-services Dockerfile and support scripts to ensure deterministic images
  • Use the CI GHA yamls and scripts to build and verify the required behavior

Tasks

Preview Give feedback
  1. Block Node Feature Enhancement Release
    mattp-swirldslabs
  2. Block Node Feature Enhancement Release
    mattp-swirldslabs
  3. Block Node Feature Enhancement Release
@AlfredoG87 AlfredoG87 self-assigned this Nov 8, 2024
@AlfredoG87 AlfredoG87 added Release Issues tied to release management functions Block Node Issues/PR related to the Block Node. Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. labels Nov 8, 2024
@ata-nas
Copy link
Contributor

ata-nas commented Nov 11, 2024

Maybe this is a duplicate of #299?

@AlfredoG87 AlfredoG87 removed their assignment Nov 22, 2024
@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

image

  • Important piece of deterministic build
  • com.hedera.gradle.java.gradle.kts

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

image

  • Must use UTF-8 encoding
  • javadoc as well in UTF-8
  • These changes should be available in the gradlew plugin available from Hiero

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

image
image
image

  • Steal this from services
  • It's used to make the jars deterministic
  • Ensures the commands we need are available
  • Hashes of the jar files, sorted list, with hash - want to use the SHA256SUM -b (binary switch)

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

image

  • Pipeline file must use UTF-8 as well
  • Steal this too

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

  • We'll need a service account for the repo
  • Might need to comment out the application section if we don't have multiple jar file directories
  • BASELINE part is important for testing the determinism

@mattp-swirldslabs
Copy link
Contributor

image

  • Standardize git line endings is also important

@mattp-swirldslabs
Copy link
Contributor

image

  • Docker determinism pieces

@mattp-swirldslabs
Copy link
Contributor

SOURCE_DATE_EPOCH="0" is not valid in later versions of docker build kit

@mattp-swirldslabs
Copy link
Contributor

image

  • Script used to make apt-get deterministic in the docker build

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

  • Might not need the "Deterministic Build Hack" section in the Dockerfile but that hasn't been verified.

@mattp-swirldslabs
Copy link
Contributor

  • S6 overlay may not be required.
  • End up with 2 layers in the end?

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Nov 26, 2024

GHA testing:
image
image

  • You can check the output of the GHA scripts to verify the run was deterministic

@ata-nas
Copy link
Contributor

ata-nas commented Dec 4, 2024

@AlfredoG87 @mattp-swirldslabs maybe #343 could be added here to be fixed together?

@mattp-swirldslabs mattp-swirldslabs self-assigned this Dec 13, 2024
@jsync-swirlds jsync-swirlds changed the title Modify Docker build to create Deterministic Images EPIC: Modify Docker build to create Deterministic Images Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Block Node Issues/PR related to the Block Node. Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. Release Issues tied to release management functions
Projects
None yet
Development

No branches or pull requests

3 participants