🚀 Cosmos Gen3 re-write proposal #3088
Comments
|
o ma gawd |
|
yeeey ilc PS: I'm writing a POC for multiplatform C# compilation using https://github.com/xiaoyuvax/MOOS/blob/master/MOOS.bflat.md. My objective is to run at least Cosmos.System under x64 and arm64 |
|
sysV abi, VFS and stuff being optional, possible x64, holy cow |
|
i have been looking in to porting cosmos plug system to work with any .net project (build time patching of .net dlls)and to make a X# emitter for NativeAOT https://gitlab.com/liquip/nativeaot-patcher |
this is incredible |
|
I was thinking on trying to implement something like the 'WebApplication' builder of ASP .NET, but for cosmos, an object where you could add the drivers and devices that needs to initialize, configure logging and maybe Dependency Injection (that is Source Generated in case of NativeAot), maybe could be done for gen3? |
I think we also need more links that redirect to websites with useful information so people can learn how to implement stuff by themselves |
what if there were plugins for jetbrains rider? it can use nuget packages |
I'd say that the goal for Gen3 would be to make Cosmos as IDE-agnostic as possible. As far as IDE integration goes, while plugins definetly would be nice, the user should be able to compile, debug, and test their Cosmos projects without an IDE at all. From my experimentation with Rider, making plugins that use its PSI layer is extremely cumbersome. You can make plugins that use the regular IntelliJ API set, with Kotlin, for example, but we definetly don't have the manpower to make and maintain plugins for different IDEs. |
I agree - but I also think that developing the educational aspect of Cosmos would be great for Gen3. There is an interest in Cosmos from developers only experienced with higher-level languages that still want to make an operating system. While I do believe that C is the best language to teach people low-level concepts (because of its relatively low amount of abstraction), telling someone "learn C" is definetly demotivating. C# offers a lower barrier of entry that could transition newcomers to C - later on, after they get more experience and gain more knowledge, they may return to Cosmos if they are looking to create a domain-specific kernel. |
|
@valentinbreiz @zarlo, is there any news from other maintainers/coredevs? Given that this would be a massive undertaking, and a huge change to the entire project, I feel like at least most of the maintainers would have to give this concept the green light. |
|
with how slow the progress already is its going to take several years to finish this sort of thing |
nothing re this, might need to give it a week |
all it takes is a big team and dedication :) |
|
what |
he is the flabbergasted |
|
We definetly appreciate the enthusiasm, but let's keep the discussion here more akin to a forum, please :) |
What I'm about to say might be a bit controversial, but from someone who has seen Cosmos mature for the past 12 years, I hate to say this but a significant amount of Cosmos users aren't very mature. I stopped using Cosmos and the discord server mainly due to the rampant racism/sexism/etc. which never seems to go away. While I love Cosmos and it's always had a place dear to me (hell, I was 12 when I first used it and now I'm 24), IMHO it's remained a research project for this reason. That being said, a rewrite/rebase would be a good idea, help bring the source into the 2020s 🙂 |
|
Wow, I didn't see that coming. You're, if not, goated. I have a question, how would we handle concepts that should be customizable but is required to run (like the bootloader for example)? We would have to quickly look into that if the project is accepted so that education-wise and for experienced users, everyone is happy. Even in it's current state, Cosmos doesn't offer proper personalization concerning config files, which, quite counter the goal of cosmos. |
I stopped using Cosmos because of just the community being 99% kids and 1% developers that actually contribute to the project, and also because I wanted to learn real OS development (and cosmos wasn't getting much progress lately either) The "what" that I said is me just being surprised there's actually still people putting effort into this |
Thanks! This depends on if we are going to use a custom kernel module loader for select modules. If we were to use a built-in loader......we would include some modules as IL assemblies directly via If we were to not include any loader......we would need to provide some way for the user to transform "static" modules to "dynamic" ones. We also would provide a NuGet package providing a pre-made kernel module loader, which could include build files to quickly create dynamic modules from plain CLR assemblies (which would be considered "static"). Using APIs from dynamically loaded modulesUsing the public API surface of a dynamically loaded module originating from a CLR assembly would be tricky - we would have to somehow defer certain relocations until we load the module in question (plain impossible with e.g. Limine), or use stubs/thunks for every method originating from a module. The stub would work similar to a relocation-based trampoline - e.g., for x86-64: _ModuleDispatchStub_ExampleMethod:
mov rax, qword _InvalidDispatchStub ; pending relocation
jmp raxAt runtime, we would replace the I feel like this would need a dedicated discussion, with the maintainers weighing in. I apologize if I sound pedantic, but I stress that maintainer activity is crucial for this to go anywhere. |
|
why not rewrite cosmos to work with NativeAOT? Zarlo already mades something to use Cosmos Plug system with NativeAOT. |
ILC is a component of Native AOT compilation. It's practically the main one - the compiler. The other components of Native AOT - mainly, the specialized versions of the standard libraries and MSBuild specification files - could also be used, but we would probably have to customize them to the point that it'd just be easier to use the components selectively rather than relying on Native AOT's regular deployment methods.
x86-64 is not only possible w/ ILC, but required for non-Windows ABI targets. There is no |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
wait, so when gen3 happens, it wont be possible to make 32-bit oses anymore? |
It technically is possible (see ascpixi/smolsharp#3), but it's very much not officially supported. I don't see why we would include support for 32-bit compilation anyways, as that is practically obsolete in the current era. |
|
a few reasons:
|
I'm not quite convinced - supporting legacy systems is always added workload on the backs of developers. re. 1. re. 2. re. 3. re. 4. Given that we still haven't heard anything concrete from any of the maintainers for a month now, I'm afraid that the only way to advance this project is with a fork, with stricter moderation, and with a more active and professional team. :( |
i meant that 32-bit oses can run on both 32-bit and 64-bit machines. imagine you want to repair some old computer using your own os, but cant because its 64-bit. i am aware you can just use something else that actually works on 32-bit machines, but what if you really want to try your own thing on practice? |
|
if we support 64 bit we will not support 32 bit (does not mean we will prevent people but you will be on your own) this goes for ARM and x86 like if you ask me base line support should be x86-64-v2 or x86-64-v3 |
|
Dude x86-64 was introduced in the late 90s and the first 64-bit x86 CPU appeared in 2003. I genuinely do not see a point in supporting i386 along with x86-64 |
Your average $7 laptop you got from your neighbor is very probably gonna be 64 bit |
That seems like an extremely specific use-case. I wouldn't recommend using Cosmos for that purpose, then - just as you shouldn't use Cosmos for e.g. creating a domain-specific OS for PA-RISC. x86 simply wouldn't be a supported architecture (x86-64 and ARM64, however, would be). It would be nice to be able to target as many architectures as possible, but as it stands now, we should target only the most important architectures first. The difference between 64-bit virtual address spaces and 32-bit ones is massive - it greatly simplifies memory management with things like a direct map. For R/W mappings without any special page table attributes, all it takes to create one is just We would be writing the code to be architecture-independent, and that includes using e.g. |
|
All these things are interesting to hear, but i think we also need some kind of structure of how gen 3 could be made, like, should x thing be done first or stuff that we should mainly focus on to discuss, etc. |
|
Another improvement that could be integrated into gen3 is an improved plug system, and in general going about plugs a different way. As it is now, cosmos is only focused on plugging the lowest level of every api. That's nice and it means a lot of things will work out of the box with no deviation from standard, but... Some things might be better to simply reimplement instead of plug to avoid what I call plug whack-a-mole. It only works until you run into the next thing that needs to be plugged. At the very least I have a suggestion to make plugs much easier to develop. Anyway, that's not to mention how the contributions might actually go about writing current plugs. Here is an example from UnmanagedMemoryStreamImpl: byte* pointer = null;
try
{
for (int I = 0; I < nInt; I++)
{
aSpan[I] = *(pointer + aStream.Position + aStream.Capacity);
}
}
catch { }...You see the problem here, right? Plug system improvementNow, how can the plug system be improved? The first thing is move away from it being a simple patch operation, and more of an assembly merge. namespace _PLUGS.System.IO {
public abstract class FileStream : Stream {
[NativePlug(/* ? */)]
public override extern void WriteByte(byte b);
public override long Seek(long offset, SeekOrigin origin) {
// ...
}
}
}Notice how this naturally expresses the members as overriding the original methods without the need for messy attributes. (Fun fact: You could also probably put this directly in System.IO too! It would work, but having to suppress the warning of a conflict would be uncomfortable) (EXTRA fun fact: This doesn't mean all the definitions have to be in the same place. If you mark the class as And this is where the implementation aspect of my proposal diverges. Instead of doing this on the fly as the project is being compiled, you would do this initial step before starting to compile the OS. As for native plugs, the merging tool could compile extern methods marked as native plugs into a stub function, which can later be linked to native code by a later step. Extra benefits of this are that the native compilation itself only has to worry about compiling the IL into native, not weaving plugs and having to compare signatures as it goes (Signatures are much easier for the merging tool to resolve as well, I personally have an issue with the cosmos plugs for FileStream.Seek failing to find their target method). 2 This also makes it easy to statically analyze what methods aren't plugged. The merge tool can generate error stubs with metadata for unplugged methods. Footnotes
|
|
Well, an improvement would be that it warns you if native code is encountered on any type that you use (as long as you are not using that member, if used, then error as usual), it could be a parameter you set on the project file or something, so that if you are working on plugin something then you get to know all of the places that possible needs patching. This could be useful also when changing .NET Versions as you can compare with an old scan to see what plugs are no longer needed and what new plugs you need to implement. I think that is may be even possible with today's implementation too. |
so a plug stub generator? if so that would not be to hard to make |
Yes |
|
are you guys actually working on this or it is just talk lol |
@Samma2009 i sent a message here with what i think its needed before coding, but nobody seems to do what's in there ;_; |
|
I mentioned this before, but first and foremost we need activity from the maintainers. We definetly would want to work on this in the CosmosOS organization, and API changes would need to be discussed. The only activity I've seen was from @zarlo's screenshot of an internal WhatsApp (?) group, but that's about it. In order to effectively coordinate this, we need:
Adding some active contributors to the maintainer team would also be welcome. There is, however, an issue of trust on this part, and so take that proposition with a grain of salt. |
|
re: the plug system improvements suggested by @RecursiveDescent In general, I agree with this new system, except for using a specialized namespace name to place all plugs into. Instead, I recommend simply marking plug classes with a using System.IO;
namespace Cosmos.Plugs;
[Plug(typeof(System.IO.FileStream))]
public class FileStream
{
// ExposeAttribute exposes private members, making a "pseudo-member". This results
// in a member that only exists as metadata - all references to it will be replaced with
// the actual member.
[Expose] ulong _privateField;
[Expose] extern void PrivateMethod();
[NativePlug(/* ? */)]
public override extern void WriteByte(byte b);
[Plug]
public override long Seek(long offset, SeekOrigin origin)
{
SomeUtilityMethod();
PrivateMethod();
// ...
}
public void SomeUtilityMethod()
{
Console.WriteLine("Hello, Cosmos!");
}
}Another benefit from generating plugged standard library assemblies ahead-of-time is better IntelliSense - methods that use unplugged methods anywhere in the chain would be dropped from the final output. I'd also like to underline that when we'll be doing plugs for the new plug system, we should never target platform-dependent methods. That is, if a method mentions either Windows or Unix, methods higher in the chain (that call such methods) should be plugged instead - even if that would result in more plugs or in duplicate code. |
Why you guys keeping the plug system.... Wouldn't it be easier to have a corelib-based system... Specially with NativeAOT Also I asked kuduzu and yes he would like to come back to cosmos in the future |
Mainly because the previous maintainers are comfortable with this system - I'm all for a custom standard library, but that will affect compatibility with existing NuGet packages (they simply wouldn't be using the same assemblies - we could make some migration tool that would automatically replace references, though). |
and it is the best way to keep dotnet updated easily. |
|
In addition to the other plug system improvements, there also should be a way to completely replace a class. Classes like After some consideration, I'm convinced that an overhaul of the plug system is indeed the best way to go for Cosmos. |
Historically, Cosmos has had 2 major revisions: gen1, and gen2. I'd like to outline the need for a new major version, and potential major improvements that can be done with a rewrite.
I'm also looking for the feedback and approval of the maintainers (coredevs).
The need for a rewrite
In the past, Cosmos has been developed with actual, standalone kernels in mind - developers could use the toolkit to quickly develop domain-specific kernels. Nowadays, Cosmos is primarily used by people wanting to learn systems programming via a high-level language.
Cosmos, in its current state, is not viable for its original use-case - that is, domain specific kernels. The reasons for this are the following:
The current development efforts focus on the wrong thing - instead of improving the core design of Cosmos, or working on its stability issues, most of the manpower is dedicated to new features.
Issues with the current code-base
A rewrite, which would reuse select components of Cosmos, would take less time than to create several, small PRs to slowly bring the codebase to an acceptable level. I'd like to highlight some of the issues with its current state, namely:
Lack of thread safety
The entirety of Cosmos isn't designed for multi-threading in mind - no code has any kind of synchronization, and some methods even use
hltto wait:Cosmos/source/Cosmos.HAL2/PIT.cs
Lines 220 to 230 in e42447d
In order to make such parts multithreaded without the use of a "big kernel lock", some portions of the project would have to be redesigned.
Poor API design
The public API surface of Cosmos could be redesigned for more extensibility, as well as to correct some OOP quirks, such as the
Deviceclass:Cosmos/source/Cosmos.HAL2/Device.cs
Lines 6 to 9 in e42447d
If the API would be largely changed, it would require all users of Cosmos to update their projects to use the new, updated API. Such breaking changes should be limited to major versions - in our case, gen 3.
Inconsistent code style and readability
There hasn't been any standardized code style for the repository, and as such, different portions of the codebase use different code styles. The most prominent example of this are some locals using system Hungarian notation, which does not benefit C# code - the types are already apparent, as opposed to e.g. C. Some other locals, however, use the .NET code style.
A large amount of code also has been designed with older C# versions in mind - resulting in the lack of use of fluent APIs and poor code quality. Principles like DRY and composition over inheritance are also often overlooked.
The public API documentation is also in a poor state - for example:
Cosmos/source/Cosmos.HAL2/BlockDevice/Partition.cs
Lines 71 to 78 in e42447d
Such poor documentation is widespread throughout the repository.
Reliance on IL2CPU
Cosmos currently heavily relies on its compiler, IL2CPU. However, it is the largest cause of odd bugs, lack of debugging support, and performance problems. It also does not follow conventions set by other compilers.
Poor Linux support
Currently, Linux support is quite poor - while one can compile Cosmos kernels under Linux, debugging them is extremely hard. On Windows, Cosmos's Visual Studio debugger is meant to solve this problem, but currently, the debugger isn't fully functional - and works on only one IDE.
Megalithic
Cosmos has no modularization. This means that every kernel, no matter if it uses it, has e.g. the network stack. Additionally, executing binaries from inside the kernel is made quite hard.
No defined ABI
While Cosmos does follow cdecl as its calling convention, it does not follow any ABI. This makes interoperability with native libraries (e.g. zlib) especially difficult.
Potential improvements in Gen3
First, let's address how some of the problems outlined in the previous sections can be avoided in the rewrite.
Design with thread safety in mind
All code that could access potentially contended resources would need to make use of some sort of synchronization - be it atomic operations, locks, etc. This could require specialized algorithms for concurrent operations.
An IPL system could also be introduced. Interrupt controllers like the APIC already support such mechanisms out-of-the-box.
Modularized approach
Instead of providing code that might not be of interest of all users to all kernels, Cosmos could split these components to NuGet packages - e.g. the network stack could be seperated.
New ILC-based compilation pipeline
This change would replace IL2CPU with ILC - the compiler used for Native AOT compilation. The pipeline would integrate with the existing plug concept.
1. Transform standard library assemblies according to the plug set
The plug set refers to a collection of assemblies with classes and methods marked with attributes recognized by the plug weaver. The plug weaver is responsible for transforming one or more target assemblies by replacing the method bodies and adding/modifying fields according to one or more plug assemblies.
Private fields and methods can be accesses from plugs via the
[Expose]attribute.2. Compile main kernel code
ILC is used to compile the main kernel code to an intermediate object file. The consuming assembly is set as the main one, and the plugged standard libraries are provided (alongside any other references defined by the author).
3. Link
Plugs should also be able to specify that methods should be included as imports. With the compilation process, the user can also specify other object files to link alongside the main one.
Note
The ability to automatically compile assembly files to object files and link them could be implemented with an optional NuGet MSBuild extension, named e.g.
Cosmos.AsmCompiler.Plugs would not provide native code directly, like it's the case now - the default MSBuild rules would include the appropriate native object file depending on the architecture, and include the right plug set for the architecture.
Addendum A. ABI
Using ILC for compilation also makes Cosmos use a popular, solid ABI - the System V ABI. This means that interop with Unix libraries is greatly simplified.
Addendum B. Compiling on non-Linux hosts
The ILC compiler emits code with a different binary format, ABI, and dependent symbols, depending on the RID. From my experience and past research,
linux-(arch)is the most documented, extensible, and solid compilation target. However, as ILC doesn't support cross-compilation, we would need authors to install tools like WSL on Windows.Addendum C. Debugging
Compiling for a Unix target also unlocks the ability to use QEMU and GDB to debug Cosmos kernels. This means that the following features would be supported with no extra effort:
Given GDB also defines a protocol, this can further be integrated with IDEs. A large amount of IDEs have GDB support.
Making Cosmos more like scaffolding, rather than a ready kernel
Currently, Cosmos defines an extremely large part of what a kernel should do. Authors that use Cosmos to learn cannot learn how to make systems like a VFS or a thread scheduler by themselves, and users already experienced with these concepts are limited by Cosmos's ready implementations. Crucial design decisions that define a kernel are already taken by Cosmos, depriving authors from choosing the design of their own operating system.
As a consequence of modularization, the base Cosmos kernel would not include all of these ready components. For those using Cosmos for education, interactive guides could be written to teach systems programming concepts. This is enticing to beginners that view lower level languages like C as "too complicated". Experienced developers, on the other hand, could write their own, custom, components.
All existing APIs, for VFS and whatnot, would be present as optional NuGet packages, for those looking to migrate or to use pre-made components.
The text was updated successfully, but these errors were encountered: