[Proposal] Reduce the workload of stdlib's maintainers with stdlib-extensions

[gabrieldemarmiesse]

This discussion is here to have a place to talk about the folloowing proposal:

• pull request
• markdown document

We are especially interested in the opinion of frequent contributors, as well as the standard library team, who are the most impacted with this change @JoeLoser @laszlokindrat @ConnorGray .

[YichengDWu]

Doesn't this require trait extentions?

In Mojo:

• Adding new functions to old structs is easy (with traits).
• Adding new old structs to old functions is easy (with traits).
• Adding new structs to old methods is hard (requires default methods/fields of traits)
• Adding new methods to old structs is impossible (requires trait extentions)

The main pain points I see are:

• Contributors need to convince the maintainers to add a new method.
• Contributors might need to convince each other for the implementation of a method.
• Maintainers need to maintain some random guys' wacky doodle methods.
• A small pain is the struct body becomes huge.

I don't see how your proposal resolves these pain points.

My proposal would be introducing trait extentions and

• The modular team have their stdlib-extentions and they own their source of truth.
• You have your own stdlib-extentions and you own your source of truth.
• Code sharing is done by default trait methods.

To which degree can your application be structured in such a way that both the data model and the set of virtual operations over it can be extended without the need to modify existing code, without the need for code repetition and without runtime type errors.
-- Mads Torgersen, The Expression Problem Revisited

[martinvuyk]

The issue is that that random guy disappears and now the maintainers need to carry the burden to maintain the code.

That is why a standard library has higher standards and shouldn't accept bad code...

everyone wants to add new methods and then the file becomes unwieldy and hard to read

Go and read some of Python's standard library code... we're still ok over here, much better I'd say.

If one wants to add a method he can just define it outside of the type in a separate package

I don't know what you are pointing out as an issue here. Right now, you can define a function that takes in a type as its input and do as any struct's internal method would... Yes you can't extend that type's signature, but you can't in Python either since you need to inherit that type or compose it inside a field of another Class. Classes are something that Modular has been clear that is in sights, but that they want to wait to build things out properly at other lower levels first.

[YichengDWu]

That is why a standard library has higher standards and shouldn't accept bad code

It's not even bad, just methods most people don't care and don't use.

Go and read some of Python's standard library code...

I did read some python code. If you happen to have read huggingface's diffusers source code you would see massive use of mixin classes and you get complex inheritance hierarchies. It's totally ok if you are fine with that. Python does not solve the expression problem per se. I would suggest you read some code from trait-based languages.

you can define a function that takes in a type as its input and do as any struct's internal method would...

I'm not sure what you mean here. You either gave up on generic code or on single dispatch.

you can't in Python either

Yes you can in Python. In case you don't know:

In [1]: class F:
   ...:     pass
   ...: 

In [2]: def foo(): print("haha")

In [3]: F.foo = foo

In [4]: F.foo()
haha

you need to inherit that type or compose it inside a field of another Class.

This is exactly the expression problem.

Classes are something that Modular has been clear that is in sights

Yes but I feel like modular favours traits more at the moment.

[martinvuyk]

Yes you can in Python. In case you don't know:

That example lives at runtime and does not extend the type's signature. And it uses dynamic Class features: read this Mojo proposal

Python does not solve the expression problem per se

What problem ? If you want to extend a type's functionality you inherit or compose or build utility functions around it, that is how programming works when using an external library. Otherwise you could have conflicts in future redefinitions.

I would suggest you read some code from trait-based languages.

Rust does not permit defining external trait implementations for foreign types: Rust's coherence rules. You have to define your own internal trait and use that, which is to say: build your own abstractions if you want to extend the type's functionality

[YichengDWu]

does not extend the type's signature.

It extends the type's signature at runtime.

In [6]: F.__dict__
Out[6]: 
mappingproxy({'__module__': '__main__',
              '__dict__': <attribute '__dict__' of 'F' objects>,
              '__weakref__': <attribute '__weakref__' of 'F' objects>,
              '__doc__': None,
              'foo': <function __main__.foo()>})

Unless you have a different definition of type signature?

that is how programming works

Bro trust me, I have heard more than a hundred times that someone tells me "this is how X works". If your mind is wired that way then I don't think we have room for discussion. But I understand where you are coming from.

[YichengDWu]

Btw Swift-like trait extensions is clearly on modular's roadmap. I link it again here.

[helehex]

I am open to trying new workflows and seeing what works best. If i'm understanding correctly, this would be like having a community library, where we can move faster and try things out collectively without putting stress on modular or breaking things that rely on the std library. If that's the case, maybe we could do it externally on our own terms? I'm also a little concerned that adding the extra layer could slow things down or cause unnecessary fragmentation or confusion, but i'm not sure.

[helehex]

i'll also mention, I can definitely see the benefit in something like this, since I (and maybe others) already build small generic libraries for projects while things are still being worked out in the actual stdlib, then once a pr goes through, i switch things over to that. It lets me move a little faster, but i could also imagine doing the same things even if there was a more lax community library, though maybe to a lesser extent

[gabrieldemarmiesse]

I think you understood well the proposal indeed. About the "on our own terms", we could do it, but if the stdlib team doesn't redirect random new features to stdlib-extensions when new contributors are coming to mojo, we'll just cause more confusion about the process.

But if there is a good consensus among the top contributors of Mojo, we can do stdlib-extensions without Modular's blessing, and then come back for their blessings later on when we proved that this system is working well. In the meantime, some new features might slip by and be integrated directly to the stdlib without going through stdlib-extensions, but I guess it's not too bad.

We can wait for more feedback and decide when we know what the top contributors think about this.

[mzaks]

I like the idea. I think an incubator might be a better name as extension might spark confusions with the technical capability of type extension. If I understand Gabriels proposal correctly, he wants to have a place where people can contribute features maybe try out different concepts which at some point can find their way into standard library, without putting an unnecessary strain on the Modulars team.
I also find it a good idea that the incubator is based on top of the stable release, not nightly.

There is a technical question of how one would be able to iterate on top of a type already present in the standard library. My guess is we either introduce wrapper classes or replicate the types in the incubator. Either way it will help with a pace of innovation and hopefully reduce the load on the Modular team (even if it is just by rejecting a PR and asking the contributor to do it in the incubator repo).

That said, we need to be aware of the fact that in some cases the development in the incubator might be contrary, redundant or misalign with the work done by Modular, so it will be common that we would need to refactor and delete some "experiment" in the incubator on each public release. This will make it hard for the third party developer to depend on the types defined in the incubator, hence an incubator based package could be a very fragile thing. We also need to consider the amount of work, which will be introduced on every Mojo stable release.

That all said, I still think that it should be a viable strategy for rapid innovation on Mojo standard library, be it about new features, performance optimisations, or experiments.

[gabrieldemarmiesse]

I think an incubator might be a better name as extension might spark confusions with the technical capability of type extension.

Totally agree.

There is a technical question of how one would be able to iterate on top of a type already present in the standard library.

We'll see on a case by case basis. When it's adding new methods (a common use case), we can just convert that into a free functions.
If it's adding/modifying attributes, it would be too much hassle for stdlib-extensions it can go in the stdlib directly. If too much work is required to make the feature live outside the stdlib, it should go in the stdlib directly. For example the short string optimization feature isn't a good fit for stdlib-extensions because it changes everything in the String struct.

This will make it hard for the third party developer to depend on the types defined in the incubator, hence an incubator based package could be a very fragile thing.

Indeed. We shouldn't advertise this package as stable.

[mzaks]

For example the short string optimization feature isn't a good fit for stdlib-extensions because it changes everything in the String struct.

I actually think it is the other way around. I think it would be better to recreate a String in the incubator with a special name and try to implement the small string optimisation there first. It is kind of what was done in C++ community AFAIK. After the incubator solution is proven and maybe even tried in some projects, it can be suggested to move to standard library, or graduate to it's own repo, in case that standard library went with another solution, but there are third parties which would rather go with the solution from the incubator.

[gabrieldemarmiesse]

Indeed this can be seen on a case by case basis and depends on the amount of additional work that one has to do to go through the incubator first. The Modular staff can also chim in and express if they want the feature quickly or not (meaning they'll allocate time to review it) and are confident about the design. Sometime it may not be 100% clear and there may be the need for a decision from the Modular staff, since they have good visibility on the internal roadmap.

[martinvuyk]

Hi, I totally support this idea. I have a repo where I've been developing some things because of reasons y'all have pointed out.

we could do it externally on our own terms?

I think that would be very detrimental to the unity of the ecosystem, we need Modular's blessing and support. The main README should redirect developers, and many other aspects that could otherwise fragment the community.

I think it would be better to recreate a String in the incubator with a special name

or we just name it the same since aliasing works pretty well and we'd use package_name.path.struct_name 🤷‍♂️

The only thing I'm a bit wary is of focusing on stable release and calling it extensions or incubator for things that are really experimental, thusly I propose we also add an experimental package inside this repo that becomes the place for things that get broken that cannot be used as a dependency outside of that sandbox package.

If we truly want to call it extensions/incubator, we need to decide if we want to take typing_extensions approach and offer backwards compatibility (latest x versions keep things in the same path, can be used with diff. main versions, etc.), or if we just treat it as only supporting (and being bound to) latest stable release, what kind of safety standards and EoL. schedule do we keep (patching old version's security vulnerabilities, mem. leaks), etc.

Having said that, I think stdlib-extensions would be a better name since I bet many things that could be added to the stdlib but have very little justification for being in a standard library (specific protocols, open standards, esoteric data structures, specialized optimizations, etc.), could very well be permanent residents there (I think SmallString would be one of those).

Another reason why I think this will be great for Mojo's long term health is that building stuff gives better insight into other use cases than only what Modular needs, and we could take some of the burden of keeping the community healthy and engaged off of their shoulders as well. And Mojo is still in it's infancy, have a look at Python's issues and PRs ... no startup company can pay enough developers to mantain that.

[gabrieldemarmiesse]

Yep sorry I forgot about the shoutout, I had the idea of this proposal after looking at your repo :)

I think that would be very detrimental to the unity of the ecosystem, we need Modular's blessing and support.

Yeah, while I believe we can do it without Modular's blessing if the top contributors are on board, it will be indeed much better if modular is on board with it.

we need to decide if we want to take typing_extensions approach and offer backwards compatibility

Not even the stdlib have backward compatibility. There is no package manager in mojo, so I don't see how backward compatibility would help anybody. The target is to have a green CI on all commits of the main branch, and that should be good enough.

what kind of safety standards and EoL. schedule do we keep (patching old version's security vulnerabilities, mem. leaks), etc

I'd say that just like in the stdlib, everything is on a "best effort" basis. Mojo is not ready for production, nobody should use it in production, I don't think we can or should provide any commitment there.

Having said that, I think stdlib-extensions would be a better name

Frankly I don't mind too much about the name of the repo. I have no strong opinion on the subject. We can revert it to stdlib-extensions if you wish. I think you can discuss that with @mzaks who has an opinion about it too.

[bgreni]

I think this could definitely be a good way to improve the velocity. My one concern is the goal of having this be more of a contributor run effort is we lack the insight on internal discourse and decision making, so the maintainers may still get frequently pinged for questions like "is this change something that would ever get accepted into the stdlib anyways?", which would kind of defeat the point. I'm just not sure we have enough visibility into the decision making process internally to reliably make decisions that are in line with the short term goals of the language as defined by Modular itself at the time. Which could end up in a significant amount of frustration and wasted effort from contributors.

[gabrieldemarmiesse]

Thanks for voicing your concerns, they're legitimate :) I'll try to address them

so the maintainers may still get frequently pinged for questions like "is this change something that would ever get accepted into the stdlib anyways?

In a typical extensions/incubator repo, the core developers rarely/never ask the opinions of the maintainers of the main repo (here modularml/mojo). The questions the core developers ask themselves when doing a code review is "Can this be a good fit for the standard library?" If the answer is "yes" or "maybe", then the feature is accepted. If the answer is "no" the feature isn't accepted. Of course maintenance cost must be taken into account.

Now comes the question "What if the core developers were wrong? And the Modular staff has a different opinion about the feature?"

Well in most cases, the Modular staff won't have an opinion in the subject, because only very few new features are needed at any given time. So the likely answer they'll give is "We will maybe accept that in the future, we'll see". The stdlib-extensions repo is a good fit for this situation as it allows one to play with the feature, reache a high quality level, judge the popularity of the feature and be just one pull request away from being in the Mojo stdlib.

Then we have the situation where the core devs accepted a feature, but the stdlib team in the end decided that they don't want it. This happens all the time, and it means that it's time for this feature to go in a separate package and repo (when we have a package manager).
It's not because it's not in the stdlib that the code has no value. So in the end no code was lost. The code is one copy-paste away from being an independant package!

The last possible difficult situation is when the core developers reject a feature, but later on, the stdlib team says they want it.
Then no harm done. The core developers will acknowledge the decision. Then a second decision must be taken by the stdlib team: do they want the feature to go through stdlib-extensions first or can the contributor add the feature directly to the stdlib? They'll decide depending on the urgency, the size, the additional work required, the complexity, etc...

In the end, I don't think any of those scenarios will create frustration. Let me know you thoughts! Note that those scenarios are based on my experience working with incubators/extensions repos in Keras and Tensorflow. We can slightly change the process if we believe it's relevant.

[bgreni]

Thanks for the thorough response! That does ease my worries a bit. I wonder if it might also be beneficial for the maintainers to keep a list of "things we want but can't prioritize internally at the moment" so we can have some sense of their current priorities as external contributors.

[gabrieldemarmiesse]

I agree with you, that would help!

[rd4com]

Hello @gabrieldemarmiesse , sorry for delay.

When mojo was newer a while ago, community did a repo for that.
then modular open-source so we can participate the effort directly.

As your proposal explains well,
employees should not have to waste time on unprioritized things.

The idea would require them to use time,
so that high quality standards is maintained in extension branch.

I think it just would be better to continue working on nightly,
modular will grow so the capacity to handle PR's too.

It does not mean that the proposal is bad.

Just that there are plenty of things to do in nightly.

We contributors also need to focus time on building nightly.

Once there are more contributors on nightly, than we can start creating extension-tasks ?

 

Your idea would jive well in the package-manager proposal.

• Scaffolding a new empty package git repo with CI, contribution guides and things like that
• The special std-contributible-effort template would create an empty environment where commits are squashable with co-authors, ...
  (it is basically an empty repo with proper contribution rules, ..)

That way, people can work on libs with confidence that it can be backported/integrated into nightly if needed.

It would also make possible to have a list of such repos for real-time considerations.

That way people can work on new packages and employees can discover and merge stuff on the fly.

[ConnorGray]

Hi Gabriel, thank you for taking the time to write up this proposal. The issues you highlight are real points of friction, and we really appreciate you taking the time to articulate what I think many folks, both internally and externally, have been feeling. I also want to shout-out the community for the robust conversation on this Discussion thread.

TL;DR: We think we should hold-off, for the time being, on implementing the new repository suggested in this proposal, but we are open to re-evaluating this down the line.

As regular contributors are aware, the standard library team has less capacity to review PRs than the community has to make contributions. To be clear, this is a good problem to have 🙂, and we're incredibly grateful to the community for the enthusiastic, high-quality, and friendly atmosphere present in the Mojo community. But as the proposal outlines, this mismatch in bandwidth can lead to delays in PR review and in answering community questions. As an aside, we're actively working now to grow the team, which should help increase our bandwidth for community work and PR reviews; nothing to report on that as of yet, but stay tuned🙂.

(If you're interested in applying to join Modular and help us out, you can find open roles on our careers page 🙂 )

In my view, the problems we're dealing with at the moment are a natural and expected (but uncomfortable) part of the process of scaling a community around an open source project—particularly an open source project that is a sensitive component of the larger MAX product Modular is building. However, despite the risks involved, we on the stdlib team, and Modular as a whole, are committed to releasing and open sourcing what we can, and continuing and building on that.

To give some context, I think it's worth taking a step back and seeing where we're at in time with regards to Mojo:

There are a few things I want to highlight:

• Mojo and the standard library especially are still very young. This is still early days for all of us working on Mojo.
• We're very early in the history of open source for the standard library. We're still learning (and sometimes making mistakes) along with the community about what works, what doesn't, and how we can most effectively use the resources we have.
• We have been consistently releasing more and more functionality over time, and our intention is to continue that trend.

We on the standard library team sometimes have a hard time believing its only been a few months since the standard library was open-sourced—the pace has been exhilarating🔥🙂

It was a large, company-wide effort to release the Mojo standard library as open source. But now, as folks are probably also aware, Modular as a whole is working hard on our upcoming release of GPU support in the MAX platform. We still have big plans for the future of open-source at Modular and for Mojo specifically, but we're hoping to keep things stable with the stdlib process for a short period while we're focused on MAX and our 🔥 GPU launch later this year.

Some thoughts on the proposal

With that general policy in mind, I do want to respond specifically to some of the points in Gabriel's proposal and some of the surrounding discussion. None of these are reasons to never implement this proposal, but they are reasons we feel being conservative about making additions to the Mojo contribution story is the right choice in the current stage of Mojo's development.

1. An additional repository is an additional layer of indirection in the new contributor experience, and in the process of getting a change merged.
2. A new repository would involve creating and maintaining several new pieces of process, including: CI testing and administrative checks, stdlib-extensions => mojo/stdlib syncing, "outbound" syncing of internal changes to the stdlib-extensions directory, and more.
3. If we were to decouple the stdlib-extensions directory in the name of keeping it light on process, that repository would often fail to build when new nightly Mojo compilers with breaking changes are released. This could lead to a worse overall experience, given the number of new features Mojo regularly gains in nightly releases, particularly for new contributors not accustomed to the special status of the 'stdlib-extensions' directory.
4. Given the proposal that the stdlib-extensions directory could be the 'first-line' place that new stdlib changes are made, Modular has a direct interest in the quality of the first-time contributor experience. That means we'd still have a direct stake, and therefore time requirements, in the new repository.
5. As pointed out in the proposal, the majority of PRs are adding features, but a large number of those PRs take the form of new methods on core standard library types like String. Because Mojo lacks support for an "extension" mechanism, currently its not possible for the definition of a type to come from multiple files. This limits the utility a stdlib-extensions repository would have to only very non-core functionality. Given that Mojo is still young, much of the "low-hanging fruit" that the open source community is most able and passionate to help with are precisely things like filling in missing methods on core types.

There very well may be ways to mitigate some of what I've mentioned here, and we're excited to see the discussion about what we should do when the time comes. But, as mentioned, our current policy feeling on this proposal is "yes, but not quite yet".

[gabrieldemarmiesse]

Big thanks to the stdlib team for taking the time to review the proposal. While I disagree with the result, I don't have all the knowledge about everything that is going on internally.

The stdlib team has good visibility on what's going on inside Modular and on the community side.
The community has good visibility on the community side but not on what's going on inside Modular.

Given that you folks have more info overall, the only correct thing to do is to trust your judgement. So while I can see how stdlib-extensions would work on our side, we can't judge the negative impact it will have internally. This proposal was kind of like shooting in the dark, it's expected that it's a miss sometime.

I hope that this can be implemented in the future as more and more things are publicly visible and open-source. When this time comes, it will be easier for the community to help out.

For the moment I'll close the pull request and the discussion. When it's time to put it on the table again, lots of things will have changed, I'll expect a brand new proposal to be done one the subject.

Cheers and best of luck on the GPU work, we can see how busy you folks are with it :)

[martinvuyk]

Some thoughts on the proposal

Thank you all for the thorough explanation and for being so open to ideas and proposals, it is very refreshing and speaks volumes of the future health and prosperity of this project 💪 .

(If you're interested in applying to join Modular and help us out, you can find open roles on our careers page 🙂 )

I would love to work with you guys. Though for various life reasons (thinking of starting a master's next year, and other stuff) and because the immigration process in the US is so damn complicated (and outdated) it isn't a possibility. If you extend to include contracting remote internationally you might get more applicants though (3rd world countries like mine are way cheaper) 👀 .

As regular contributors are aware, the standard library team has less capacity to review PRs than the community has to make contributions.

Yeah sorry hehe

TL;DR: We think we should hold-off, for the time being, on implementing the new repository suggested in this proposal, but we are open to re-evaluating this down the line.

I'll be a bit frank here. I know Modular is currently at the very early stages of a startup, and profitability is not in scope yet. But just doing some quick math having 8 US developers mantaining a repo with similar activity (even with 1/3 is a lot) to Python's repo will take more than 1M dollars a year and it won't be enough. You guys have the clock ticking to setup how Mojo will transition to fully OS, otherwise it will bankrupt you and the community will get too impatient with PRs taking so long and it feeling like you are doing free work for a company's internal tooling. And I think I speak for many by saying that I'd like this project to prosper beyond Modular's lifetime if it falls short, because I think it will benefit everyone greatly.

We on the standard library team sometimes have a hard time believing its only been a few months since the standard library was open-sourced—the pace has been exhilarating🔥🙂

Having said that I wanted to end on a more positive note. What we are building together is awesome in every sense of the word, thank you for making it open source and having the patience and openness to deal with the community ❤️