image_partition: add ability to skip "start" and specify relative offsets #285

thanhtunng · 2021-08-27T10:44:02Z

Currently start and end value must be specified explicitly

We can ignore start value unless required by using the end of
previous partition.

The size of partition can also be easier to see if using
end value as relative offset (e.g: +10Mib, +5%). However,
the type of end value must be the same as start's

Example:

- action: image-partition
  imagename: "debian-rpi3.img"
  imagesize: 1GB
  partitiontype: msdos
  mountpoints:
    - mountpoint: /
      partition: root
    - mountpoint: /boot/firmware
      partition: firmware
      options: [ x-systemd.automount ]
  partitions:
    - name: firmware
      fs: vfat
      end: +5%
    - name: root
      fs: ext4
      start: 64MB
      end: +256MB
      flags: [ boot ]

firmware partition - start: 0% (as default), end: 5%
root partition - start: 64MB. end: 320MB

sangorrin · 2021-09-12T13:59:03Z

Hi,

Just to make sure, this is not an essential feature for us.
I just thought the interface would be more user-friendly if we used relative offsets.
Perhaps there are other users interested.

Thanks,
Daniel

evelikov · 2021-10-26T21:28:37Z

@thanhtunng @sangorrin thanks for the neat feature - was looking to have start optional.

The MR does 3 things (see below) yet it's a single commit - do you mind splitting it up? Mind you I don't have commit access to the repo, so this is just a humble request. Thanks o/

makes start optional
adds basic validation for start and end
adds relative offset support for end

We can ignore start value unless required by using the end of previous partition. Signed-off-by: Vu Thanh Tung <[email protected]>

The size of partition can also be easier to see if using end value as relative offset (e.g: +10Mib, +5%). However, the type of end value must be the same as starts. Signed-off-by: Vu Thanh Tung <[email protected]>

Signed-off-by: Vu Thanh Tung <[email protected]>

thanhtunng · 2021-10-27T16:34:39Z

@evelikov thanks for your suggestion, i changed follows yours. Please let me know if you need any improvements.

evelikov · 2021-10-27T22:56:05Z

Looks great thank you.

In case I was ambiguous earlier - I cannot merge this MR. since I do not have commit access.

evelikov · 2022-01-21T12:54:25Z

Tested the MR and it seems kind of broken. So it needs extra work until it's in good state.

Example:

architecture: amd64

actions:
  - action: image-partition
    imagename: disk.img
    imagesize: 2G
    partitiontype: gpt
    partitions:
      - name: rootfs
        fs: ext4
        end: +512Mi
      - name: home
        fs: ext4
        end: 100%
    mountpoints:
      - mountpoint: /
        partition: rootfs
      - mountpoint: /home
        partition: home

Issues:

rootfs start/end unit mismatch - trivial fix, ignore unit when startval == 0
creating the home partition outright fails

$ parted -a none -s -- /dev/disk/by-fakemachine-label/fakedisk-0 mkpart home ext4 512Mi 100%
Error: You requested a partition from 537MB to 2000MB (sectors 1048576..3906249)
The closest location we can manage is 537MB to 2000MB (sectors 1048577..3906216)

evelikov · 2022-01-21T12:57:14Z

actions/image_partition_action.go

+		case l >= 'A' && l <= 'Z', l >= 'a' && l <= 'z', l == '%':
+			t = append(t, l)
+		}
+	}


This loop here allows for arbitrary alpha-numeric word soup. For example "A01Z99%" is considered sane by the above.
Let's use a regex akin to the one in #292.

@thanhtunng Any update?

evelikov · 2022-01-21T12:58:42Z

actions/image_partition_action.go

+
+func CalculateOffset(start, end string) (string, error) {
+	/* Do addition if end value uses a '+' prefix */
+	if strings.HasPrefix(end, "+") {


Nit: I would move the "HasPrefix" check in the caller.

thanhtunng · 2022-01-23T17:24:56Z

rootfs start/end unit mismatch - trivial fix, ignore unit when startval == 0

@evelikov Yeah the mismatch issue I already mentioned in commit message, anw the case startval ==0 I didnt think about, thanks~

$ parted -a none -s -- /dev/disk/by-fakemachine-label/fakedisk-0 mkpart home ext4 512Mi 100%
Error: You requested a partition from 537MB to 2000MB (sectors 1048576..3906249)
The closest location we can manage is 537MB to 2000MB (sectors 1048577..3906216)

MiB? Furthermore not sure able to use "MiB" unit unless #292 is merged

evelikov · 2022-03-05T12:30:15Z

You might be conflating things:
Mi/MiB handling with partitions just works in the master branch, since the raw data from the yaml is passed to directly into parted as seen in the example above. This MR breaks that functionality and I think you should look into why.

The code in #292 covers the imagesize, which is a completely orthogonal thing.

obbardc · 2022-10-12T19:02:42Z

@thanhtunng Do you have an update about this?

evelikov · 2023-01-09T13:04:09Z

@obbardc I'm keen on pushing this forward. Is there any particular changes you'd like to see (aka anything more than what I've raised already)? I'm thinking about adding some tests to the CI - the snippet above + another one with explicit start and other units.

Any pointers how to properly fix the parted error mentioned before - #285 (comment). It seems like we're either hitting a parted bug or an off-by-one error in debos.

obbardc · 2023-01-09T13:17:09Z

Proper in-depth automated tests for both relative & non-relative cases would be great with various line endings etc. I was thinking something along the lines of manually creating a partition table and comparing that at runtime.

Happy to start small though ;-).

obbardc · 2023-07-26T15:19:00Z

@thanhtunng @evelikov Any update here ? It'd be great to get this reworked.

evelikov · 2023-07-27T08:20:58Z

Nothing from me since debos maintainership has been MIA over the last 6 months or so. Will be quite busy in the upcoming 2-3 months so it's unlikely I'll have the time, sorry.

obbardc

We are missing some tests & fixing some nits. I'd be happy to merge once the above is complete.

obbardc · 2023-12-05T13:07:49Z

actions/image_partition_action.go

@@ -80,6 +75,11 @@ GPT sets the partition type to Linux Swap.
 For msdos partition types hex codes see: https://en.wikipedia.org/wiki/Partition_type
 For gpt partition type GUIDs see: https://systemd.io/DISCOVERABLE_PARTITIONS/

+- start -- offset from beginning of the disk there the partition starts.


It'd be good to note why this is optional and what would happen if it isn't actually provided.

obbardc · 2023-12-05T13:08:48Z

actions/image_partition_action.go

@@ -80,6 +75,11 @@ GPT sets the partition type to Linux Swap.
 For msdos partition types hex codes see: https://en.wikipedia.org/wiki/Partition_type
 For gpt partition type GUIDs see: https://systemd.io/DISCOVERABLE_PARTITIONS/

+- start -- offset from beginning of the disk there the partition starts.
+
+For 'start' and 'end' properties offset can be written in human readable


It'd be good to duplicate this inline so that both start and end properties have this.

obbardc · 2023-12-05T13:10:03Z

actions/image_partition_action.go

@@ -642,12 +643,14 @@ func (i *ImagePartitionAction) Verify(context *debos.DebosContext) error {
 		}

 		if p.Start == "" {
-			return fmt.Errorf("Partition %s missing start", p.Name)
+			p.Start = prevEnd


It'd be good to move this logic into the Run stage rather than Verify stage potentially with some comment about what is going on.

obbardc · 2023-12-05T13:10:37Z

actions/image_partition_action.go

@@ -642,12 +643,14 @@ func (i *ImagePartitionAction) Verify(context *debos.DebosContext) error {
 		}

 		if p.Start == "" {
-			return fmt.Errorf("Partition %s missing start", p.Name)
+			p.Start = prevEnd
 		}
 		if p.End == "" {


This really should have a test case as well.

obbardc · 2023-12-05T13:10:59Z

actions/image_partition_action.go

+		case l >= 'A' && l <= 'Z', l >= 'a' && l <= 'z', l == '%':
+			t = append(t, l)
+		}
+	}


@thanhtunng Any update?

obbardc · 2023-12-05T13:11:20Z

actions/image_partition_action.go

+
+func CalculateOffset(start, end string) (string, error) {
+	/* Do addition if end value uses a '+' prefix */
+	if strings.HasPrefix(end, "+") {


obbardc · 2023-12-05T13:11:52Z

actions/image_partition_action.go

@@ -600,6 +646,7 @@ func (i *ImagePartitionAction) Verify(context *debos.DebosContext) error {
 	num := 1
 	prevEnd := "0%"
 	for idx, _ := range i.Partitions {
+		var err error


you can remove this by having later on p.End, err := blah

obbardc · 2023-12-05T13:12:06Z

actions/image_partition_action.go

@@ -649,6 +696,11 @@ func (i *ImagePartitionAction) Verify(context *debos.DebosContext) error {
 			return fmt.Errorf("Partition %s missing end", p.Name)
 		}

+		p.End, err = CalculateOffset(p.Start, p.End)


as above, check here if prefix is +

obbardc · 2023-12-05T13:12:47Z

actions/image_partition_action.go

+		return err
+	}
+	if typ == "%" && val > 100 {
+		return fmt.Errorf("Size can not exceed 100%%")


nit: two %%

sjoerdsimons · 2024-01-13T19:36:25Z

actions/image_partition_action.go

-For 'start' and 'end' properties offset can be written in human readable
-form -- '32MB', '1GB' or as disk percentage -- '100%'.
+- end -- offset from beginning of the disk there the partition ends or
+offset from beginning of where the partition starts specified with prefix "+"


Rather then having end be either end point on the disk or the size of the disk it would make more sense to have a seperate size property and require either end or size to be provided.

size should be in similar to the units parted accepts (https://www.gnu.org/software/parted/manual/html_node/unit.html) ideally; The image size parsing is mostly there, only % and s really sould be handled. I don't think anyone still cares about cylinders :).

same parsing should probably be use for start and end rather then the current state were we just pass on whatever the user gives to parted and hope :)..

That way it's both more clear for end-user, cleanup one of the debos oddities and more flexible then this implementation where there seems to be very specific requirements between the start and end value "types"

Vu Thanh Tung added 3 commits October 27, 2021 23:11

image_partition: make "start" value optional

7335a24

We can ignore start value unless required by using the end of previous partition. Signed-off-by: Vu Thanh Tung <[email protected]>

image_partition: add relative offset support for "end" value

2328b3f

The size of partition can also be easier to see if using end value as relative offset (e.g: +10Mib, +5%). However, the type of end value must be the same as starts. Signed-off-by: Vu Thanh Tung <[email protected]>

image_partition: add basic percentage validation for start and end

9e4f3d8

Signed-off-by: Vu Thanh Tung <[email protected]>

thanhtunng force-pushed the tsdv/relative-offset branch from 597b571 to 9e4f3d8 Compare October 27, 2021 16:26

evelikov reviewed Jan 21, 2022

View reviewed changes

obbardc added this to the v1.1.2 milestone Oct 12, 2022

evelikov mentioned this pull request Dec 5, 2023

Blockers for using upstream debos #456

Closed

obbardc requested changes Dec 5, 2023

View reviewed changes

obbardc modified the milestones: v1.1.2, v1.1.4 Jan 10, 2024

sjoerdsimons requested changes Jan 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

image_partition: add ability to skip "start" and specify relative offsets #285

image_partition: add ability to skip "start" and specify relative offsets #285

thanhtunng commented Aug 27, 2021

sangorrin commented Sep 12, 2021

evelikov commented Oct 26, 2021 •

edited

Loading

thanhtunng commented Oct 27, 2021

evelikov commented Oct 27, 2021

evelikov commented Jan 21, 2022

evelikov Jan 21, 2022

obbardc Dec 5, 2023

evelikov Jan 21, 2022

obbardc Dec 5, 2023

thanhtunng commented Jan 23, 2022 •

edited

Loading

evelikov commented Mar 5, 2022

obbardc commented Oct 12, 2022

evelikov commented Jan 9, 2023

obbardc commented Jan 9, 2023

obbardc commented Jul 26, 2023

evelikov commented Jul 27, 2023

obbardc left a comment

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

obbardc Dec 5, 2023

sjoerdsimons Jan 13, 2024

image_partition: add ability to skip "start" and specify relative offsets #285

Are you sure you want to change the base?

image_partition: add ability to skip "start" and specify relative offsets #285

Conversation

thanhtunng commented Aug 27, 2021

sangorrin commented Sep 12, 2021

evelikov commented Oct 26, 2021 • edited Loading

thanhtunng commented Oct 27, 2021

evelikov commented Oct 27, 2021

evelikov commented Jan 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thanhtunng commented Jan 23, 2022 • edited Loading

evelikov commented Mar 5, 2022

obbardc commented Oct 12, 2022

evelikov commented Jan 9, 2023

obbardc commented Jan 9, 2023

obbardc commented Jul 26, 2023

evelikov commented Jul 27, 2023

obbardc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evelikov commented Oct 26, 2021 •

edited

Loading

thanhtunng commented Jan 23, 2022 •

edited

Loading