[stdlib] Make trait `CollectionElement` inherit from `ExplicitlyCopyable` #3153

gabrieldemarmiesse · 2024-06-30T10:38:31Z

Waiting for the new nightly to be released so I can rebase this PR and shrink the diff

List of issues I have currently:

1 - Passing functions by parameter gives cryptic error message

Here is a minimal reproducer:

fn foo(inout buff: UnsafePointer[Int], len: Int):

    @parameter
    fn _less_than_equal(lhs: Int, rhs: Int) -> Bool:
        return lhs <= rhs

    bar[type=Int, cmp_fn=_less_than_equal](buff, len)

@always_inline
fn bar[
    type: CollectionElement, cmp_fn: fn (type, type) capturing -> Bool
](array: UnsafePointer[type], size: Int):
    ...

The error message is

invalid call to 'bar': callee parameter 'cmp_fn' has 'fn(Int, Int) capturing -> Bool' type, but value has type 'fn(lhs: Int, rhs: Int) capturing -> Bool'

The error message is not clear at all. What can we do to fix it? This issue prevents me from migrating over sort.mojo.

2 - The `None` constructor of `Optional` cannot be used anymore. We must use `Optional[...]()` or `Optional[...](_NoneType)`

It's still present in the code, but if one tries to use it, then the compiler says "cannot bind MLIR type None to trait CollectionElement". And the error message is often at the wrong place in the code (compiler bug?)

3 - Using `Dict[Self, Self]` in `PythonObject` gives `recursive reference to declaration`

So I had to comment out the method. I don't see how to reintroduce it.

Signed-off-by: gabrieldemarmiesse <[email protected]>

…th_collectionelementnew

…_with_collectionelementnew

…h_collectionelementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

…th_collectionelementnew

…_list_set_work_with_collectionelementnew

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse · 2024-06-30T13:57:36Z

stdlib/src/builtin/format_int.mojo

@@ -330,7 +332,7 @@ fn _try_write_int[
            len=1,
        )
        fmt.write_str(zero)
-        return


Here is an example of the None constructor of Optional not working anymore.

gabrieldemarmiesse · 2024-06-30T13:58:45Z

stdlib/src/python/object.mojo

+    # TODO: re-enable when this is possible
+    # fn __init__(inout self, value: Dict[Self, Self]):
+    #    """Initialize the object from a dictionary of PythonObjects.
+    #
+    #    Args:
+    #        value: The dictionary value.
+    #    """
+    #    var cpython = _get_global_python_itf().cpython()
+    #    self.py_object = cpython.PyDict_New()
+    #    for entry in value.items():
+    #        var result = cpython.PyDict_SetItem(
+    #            self.py_object, entry[].key.py_object, entry[].value.py_object
+    #        )


Uncommenting this gives "recursive reference to declaration"

…ementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse · 2024-07-04T20:47:55Z

@ConnorGray would that help if I rebase this PR or do you have an internal branch for this that you're taking care or? (in which case I can just close this PR then)

Signed-off-by: gabrieldemarmiesse <[email protected]>

…_collectionelementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse · 2024-07-11T12:00:19Z

stdlib/test/collections/test_optional.mojo

@@ -151,10 +151,12 @@ def test_optional_equality():
    assert_true(o == 10)
    assert_true(o != 11)
    assert_true(o != n)
-    assert_true(o != None)
+    # TODO: Find a way to allow "o != None" again.
+    assert_true(o.__ne__(None))


in the meantime we can recommend is None which works well.

I think this will be fixed when we change the compiler to emit NoneType() when it sees None. (Currently it emits the #kgen.none raw MLIR type, which means there's always one extra "hop" of conversion where NoneType is currently used.)

gabrieldemarmiesse · 2024-07-11T12:01:28Z

@ConnorGray you can take a look at what this PR to get an idea of what it takes to make this work. But I'll likely split it further.

ConnorGray · 2024-07-11T13:49:35Z

stdlib/src/collections/optional.mojo

@@ -193,6 +204,17 @@ struct Optional[T: CollectionElement](
            return False
        return not rhs

+    fn __ne__(self, rhs: NoneType) -> Bool:


Suggestion I think opt == None will work if you change this signature to:

# TODO(MSTDL-715): # This method should not be necessary, we should need # only the comparison from a `Optional(None)`. fn __ne__(self, rhs: NoneType._mlir_type):

(See the similar comment and pattern on Optional.__init__(inout self, value: NoneType._mlir_type).)

I just tried and got the following error:

/projects/open_source/mojo/stdlib/test/collections/test_optional.mojo:148:5: error: cannot bind MLIR type 'None' to trait 'CollectionElement' def test_optional_equality(): ^ /projects/open_source/mojo/stdlib/src/builtin/value.mojo:148:8: note: MLIR type cannot satisfy required trait function here fn __init__(inout self, *, other: Self): ^ /projects/open_source/mojo/stdlib/test/collections/test_optional.mojo:148:5: error: cannot bind MLIR type 'None' to trait 'CollectionElement' def test_optional_equality(): ^ /projects/open_source/mojo/stdlib/src/builtin/value.mojo:148:8: note: MLIR type cannot satisfy required trait function here fn __init__(inout self, *, other: Self): ^

Since I get the same error when I didn't do any modifications to Optional I believe that it's a bug in the overload resolution.

ConnorGray · 2024-07-11T13:53:10Z

stdlib/test/test_utils/types.mojo

+# TODO: Pass directly a UnsafePointer when possible, instead of an Int.
+# Otherwise we get "argument #2 cannot be converted from 'UnsafePointer[List[Int], 0, 0]' to 'UnsafePointer[List[Int], 0, 0]'"
+# This bug also appears if we pass the list as borrow and grab the address in the constructor, in
+# which case we get "argument #2 cannot be converted from 'List[Int]' to 'List[Int]'"


Question Does this work if you use rebind[UnsafePointer[List[Int]]](ptr) wherever the pointer is passed? That might be a simpler workaround (if it does work).

I'm not sure I understand where the rebind should go, but I tried this:

@value struct ValueDestructorRecorder(CollectionElement): var value: Int var destructor_counter: UnsafePointer[List[Int]] fn __init__(inout self, value: Int, destructor_counter: UnsafePointer[List[Int]]): self.value = value self.destructor_counter = rebind[UnsafePointer[List[Int]]](destructor_counter) fn __init__(inout self, *, other: Self): self.value = other.value self.destructor_counter = other.destructor_counter fn __del__(owned self): self.destructor_counter[].append(self.value)

And I get the following error:

/projects/open_source/mojo/stdlib/test/memory/test_maybe_uninitialized.mojo:26:32: error: no matching function in initialization ValueDestructorRecorder( ~~~~~~~~~~~~~~~~~~~~~~~^ /projects/open_source/mojo/stdlib/test/test_utils/types.mojo:120:8: note: candidate not viable: argument #2 cannot be converted from 'UnsafePointer[List[Int], 0, 0]' to 'UnsafePointer[List[Int], 0, 0]' fn __init__(inout self, value: Int, destructor_counter: UnsafePointer[List[Int]]): ^ /projects/open_source/mojo/stdlib/test/test_utils/types.mojo:124:8: note: candidate not viable: missing 1 required keyword-only argument: 'other' fn __init__(inout self, *, other: Self): ^ mojo: error: failed to parse the provided Mojo source module

and if I rebind an Int, I just get

/projects/open_source/mojo/stdlib/src/builtin/rebind.mojo:42:52: error: invalid rebind between two unequal types: index to !kgen.pointer<none>

This pull request is part of #3153 and can be merged before. It's basically adding a few explicit copy constructors where necessary. I didn't add any traits, as `CollectionElement` will very soon require the explicit copy constructor. Co-authored-by: Gabriel de Marmiesse <[email protected]> Closes #3217 MODULAR_ORIG_COMMIT_REV_ID: 5593d75e811050ee28b5d981ab93ee6f789cdec2

…ementnew

ConnorGray · 2024-07-11T15:54:40Z

!sync

gryznar · 2024-07-11T17:17:36Z

stdlib/src/python/object.mojo

+    #    self.py_object = cpython.PyDict_New()
+    #    for entry in value.items():
+    #        var result = cpython.PyDict_SetItem(
+    #            self.py_object, entry[].key.py_object, entry[].value.py_object


Should this code be commented without todo / fixme etc?

This won't go into nightly, it's there to show what is needed to make the CI happy

gryznar · 2024-07-11T17:18:03Z

stdlib/src/python/python.mojo

+    #    Returns:
+    #        The constructed empty Python dictionary.
+    #    """
+    #    return PythonObject(Dict[PythonObject, PythonObject]())


Same question, is commenting that intentional?

ConnorGray · 2024-07-11T19:20:06Z

!sync

…ementnew

…ementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

…ementnew

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse · 2024-07-21T18:23:31Z

@ConnorGray I found a workaround for the Dict[PythonOject, PythonObject]. Let me know what you think :)

I believe the PR is good enough to be merged.

JoeLoser · 2024-07-21T20:48:26Z

!sync

…ementnew

gabrieldemarmiesse · 2024-08-15T14:27:32Z

Feel free to ping me when you have time for a review and want me to rebase the PR

This pull request is part of #3153 and can be merged before. It's basically adding a few explicit copy constructors where necessary. I didn't add any traits, as `CollectionElement` will very soon require the explicit copy constructor. Co-authored-by: Gabriel de Marmiesse <[email protected]> Closes #3217 MODULAR_ORIG_COMMIT_REV_ID: 5593d75e811050ee28b5d981ab93ee6f789cdec2

gabrieldemarmiesse added 18 commits June 28, 2024 11:00

[stdlib] Add trait CollectionElementNew to a few structs

ad0bc0e

Signed-off-by: gabrieldemarmiesse <[email protected]>

[stdlib] Add CollectionElementNew trait to Dict and DictEntry

fe96ef8

Signed-off-by: gabrieldemarmiesse <[email protected]>

[stdlib] Add the trait CollectionElementNew to List

a1a4785

Signed-off-by: gabrieldemarmiesse <[email protected]>

[stdlib] Add CollectionElementNew to new traits

3b98599

Signed-off-by: gabrieldemarmiesse <[email protected]>

Add unit test for legacy pointer

0d2341c

Signed-off-by: gabrieldemarmiesse <[email protected]>

Add test for arc

c7a6801

Signed-off-by: gabrieldemarmiesse <[email protected]>

Add the test for pointer copy

1a1813a

Signed-off-by: gabrieldemarmiesse <[email protected]>

[stdlib] Add CollectionElementNew to a few structs, part 3

87872f0

Signed-off-by: gabrieldemarmiesse <[email protected]>

add the traits explicitly

af298fd

Signed-off-by: gabrieldemarmiesse <[email protected]>

[stdlib] Add CollectionElementNew to a few structs, part 4

95e5b79

Signed-off-by: gabrieldemarmiesse <[email protected]>

Merge branch 'moar_copy_constructors' into make_dict_list_set_work_wi…

4bcd776

…th_collectionelementnew

Merge branch 'moar_collectionelementnew' into make_dict_list_set_work…

54ec253

…_with_collectionelementnew

Merge branch 'more_explicit_copy_traits' into make_dict_list_set_work…

06d075e

…_with_collectionelementnew

Merge branch 'add_copy_constructors' into make_dict_list_set_work_wit…

a517dd4

…h_collectionelementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

Merge branch 'more_copy_constructors' into make_dict_list_set_work_wi…

22ead11

…th_collectionelementnew

Merge branch 'give_collectionelementnew_trait_to_list' into make_dict…

4991574

…_list_set_work_with_collectionelementnew

I only need to fix sort next

f8457a3

Signed-off-by: gabrieldemarmiesse <[email protected]>

Fix most of the sort errors

336b7e7

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse changed the title ~~[stdlib] Add trait CollectionElement inherit from ExplicitlyCopyable~~ [stdlib] Make trait CollectionElement inherit from ExplicitlyCopyable Jun 30, 2024

gabrieldemarmiesse commented Jun 30, 2024

View reviewed changes

gabrieldemarmiesse added 2 commits July 1, 2024 22:33

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

adc370b

…ementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

Removed diff

a81fe92

Signed-off-by: gabrieldemarmiesse <[email protected]>

ConnorGray self-assigned this Jul 2, 2024

gabrieldemarmiesse added 3 commits July 11, 2024 11:47

[stdlib] All done with a few bugs

e326324

Signed-off-by: gabrieldemarmiesse <[email protected]>

Merge branch 'explicit_copy_take_2' into make_dict_list_set_work_with…

3c16445

…_collectionelementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

Reduce diff in types.mojo

1f5edef

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse commented Jul 11, 2024

View reviewed changes

gabrieldemarmiesse marked this pull request as ready for review July 11, 2024 12:45

gabrieldemarmiesse requested a review from a team as a code owner July 11, 2024 12:45

gabrieldemarmiesse mentioned this pull request Jul 11, 2024

[stdlib] Add a few explicit copy constructors #3217

Closed

ConnorGray reviewed Jul 11, 2024

View reviewed changes

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

8c65ce1

…ementnew

modularbot added the imported-internally Signals that a given pull request has been imported internally. label Jul 11, 2024

gryznar reviewed Jul 11, 2024

View reviewed changes

gabrieldemarmiesse added 7 commits July 15, 2024 14:19

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

1f37b1c

…ementnew

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

a89778e

…ementnew Signed-off-by: gabrieldemarmiesse <[email protected]>

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

f18f72a

…ementnew

Fix recursive reference to self issue

30c3d62

Signed-off-by: gabrieldemarmiesse <[email protected]>

Revert the contrustor move

57c22d6

Signed-off-by: gabrieldemarmiesse <[email protected]>

Better comments

95a1f36

Signed-off-by: gabrieldemarmiesse <[email protected]>

Rewording

954981c

Signed-off-by: gabrieldemarmiesse <[email protected]>

gabrieldemarmiesse requested a review from ConnorGray July 21, 2024 18:23

modular-automation bot assigned JoeLoser Jul 21, 2024

gabrieldemarmiesse mentioned this pull request Jul 22, 2024

[stdlib] Remove Copyable from CollectionElement #3282

Draft

gabrieldemarmiesse added 4 commits July 27, 2024 22:39

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

4891d5f

…ementnew

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

172f5c0

…ementnew

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

46a2475

…ementnew

Merge branch 'nightly' into make_dict_list_set_work_with_collectionel…

641110f

…ementnew

JoeLoser added the blocked Blocked by another issue that must be resolved first label Sep 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[stdlib] Make trait `CollectionElement` inherit from `ExplicitlyCopyable` #3153

[stdlib] Make trait `CollectionElement` inherit from `ExplicitlyCopyable` #3153

gabrieldemarmiesse commented Jun 30, 2024 •

edited

Loading

gabrieldemarmiesse Jun 30, 2024

gabrieldemarmiesse Jun 30, 2024

gabrieldemarmiesse commented Jul 4, 2024

gabrieldemarmiesse Jul 11, 2024

ConnorGray Jul 11, 2024

gabrieldemarmiesse commented Jul 11, 2024

ConnorGray Jul 11, 2024 •

edited

Loading

gabrieldemarmiesse Jul 11, 2024

ConnorGray Jul 11, 2024

gabrieldemarmiesse Jul 11, 2024 •

edited

Loading

ConnorGray commented Jul 11, 2024

gryznar Jul 11, 2024

gabrieldemarmiesse Jul 11, 2024

gryznar Jul 11, 2024

ConnorGray commented Jul 11, 2024

gabrieldemarmiesse commented Jul 21, 2024 •

edited

Loading

JoeLoser commented Jul 21, 2024

gabrieldemarmiesse commented Aug 15, 2024

[stdlib] Make trait CollectionElement inherit from ExplicitlyCopyable #3153

Are you sure you want to change the base?

[stdlib] Make trait CollectionElement inherit from ExplicitlyCopyable #3153

Conversation

gabrieldemarmiesse commented Jun 30, 2024 • edited Loading

1 - Passing functions by parameter gives cryptic error message

2 - The None constructor of Optional cannot be used anymore. We must use Optional[...]() or Optional[...](_NoneType)

3 - Using Dict[Self, Self] in PythonObject gives recursive reference to declaration

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabrieldemarmiesse commented Jul 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabrieldemarmiesse commented Jul 11, 2024

ConnorGray Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabrieldemarmiesse Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

ConnorGray commented Jul 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ConnorGray commented Jul 11, 2024

gabrieldemarmiesse commented Jul 21, 2024 • edited Loading

JoeLoser commented Jul 21, 2024

gabrieldemarmiesse commented Aug 15, 2024

[stdlib] Make trait `CollectionElement` inherit from `ExplicitlyCopyable` #3153

[stdlib] Make trait `CollectionElement` inherit from `ExplicitlyCopyable` #3153

gabrieldemarmiesse commented Jun 30, 2024 •

edited

Loading

2 - The `None` constructor of `Optional` cannot be used anymore. We must use `Optional[...]()` or `Optional[...](_NoneType)`

3 - Using `Dict[Self, Self]` in `PythonObject` gives `recursive reference to declaration`

ConnorGray Jul 11, 2024 •

edited

Loading

gabrieldemarmiesse Jul 11, 2024 •

edited

Loading

gabrieldemarmiesse commented Jul 21, 2024 •

edited

Loading