Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add or document a way to write unit tests for Godot C# projects (e.g. with nunit3) #432

Open
dsge opened this issue Feb 2, 2020 · 52 comments
Assignees

Comments

@dsge
Copy link

dsge commented Feb 2, 2020

Describe the project you are working on:

A 3D spaceship/spacebase building game (think Space Engineers)

Describe the problem or limitation you are having in your project:

The more code you write to do custom calculations the more cumbersome it becomes to keep looking back to see if previously written code still functions as expected. Adding more code or refactoring the current code is also much easier if you can build on properly (unit)tested foundations.

As an actual example from my current project, I have to calculate how fast my spaceship can accelerate/decelerate given:

  • the current power level of the spaceship
  • thrusters power consumption
  • number of thrusters
  • thrusters maximum performance in a given direction
  • ...and possibly more factors

Manually testing all of these again and again (and again) is possible, however it's time consuming and inefficient.

Describe how this feature / enhancement will help you overcome this problem or limitation:

Proper unittests can save a lot of time by quickly telling me if my (new or old) code still functions as expected, thus saving me time that can be instead spent on new features.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

While having a GUI is nice and all, I think CLI support is the more critical for this particular feature. Running the tests locally and in CI can eliminate further bugs while also allowing for better cooperation on the project with other contibutors.

Workflow in case it is possible to run tests directly using nunit3-console.exe:

  • (build the game dll using Godot editor)
  • ./path/to/nunit3-console.exe /path/to/game.dll --where "<filter for testcases>" --pause
    • the where option allows to filter for testcases (for when you only want to run specific ones instead of all of them)
    • the pause option allows to attach external debuggers to debug test cases
  • nunit3-console.exe outputs the tests results and returns 0 if all tests passed (for CI support)

Workflow in case the tests need to be ran through the Godot editor's CLI:

  • (build the game using Godot editor)
  • ./path/to/Godot_v3.2-stable_yourplatform --path "/path/to/game" --run-nunit3-console --where "<filter for testcases>" --pause
    • the where option allows to filter for testcases (for when you only want to run specific ones instead of all of them)
    • the pause option allows to attach external debuggers to debug test cases
  • Godot_v3.2-stable_yourplatform outputs the tests results provided by nunit3-console.exe and returns 0 if all tests passed (for CI support)

Describe implementation detail for your proposal (in code), if possible:

Unfortunately I only have limited understanding on Godot's source code or how c# (mono) support is implemented, so I do not know what needs to be implemented or changed to allow this (if anything). If using nunit3 is already possible, then this feature request is actually a request for documentation/examples.

What I do know however is that right now if you try to run your game dll with nunit3-console.exe your testcases will actually work until your testcase reaches the first call to Godot (more specifically to anything inside GodotSharp.dll). You will be bombarded with error messages like this:

cant resolve internal call to "Godot.NativeCalls::godot_icall_Camera_Ctor" (tested without signature also)

Your mono runtime and class libraries are out of sync.
The out of sync library is: /app/.mono/temp/bin/Tools/GodotSharp.dll

When you update one from git you need to update, compile and install
the other too.
Do not report this as a bug unless you're sure you have updated correctly:
you probably have a broken mono install.
If you see other errors or faults after this message they are probably related
and you need to fix your mono install first.

What this error message actually means (despite what it says) is that the c# function called Godot.NativeCalls::godot_icall_Camera_Ctor is defined as an extern function in GodotSharp.dll but no actual implementation is found anywhere. The same code (same project) does work as expected when exporting/running it through Godot Editor.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

Not that I am aware of.

Is there a reason why this should be core and not an add-on in the asset library?:

I think at least documentation on this should be included within Godot's official documentation, even if any implementation is outside of the main source code. The reason for this is to avoid confusion by providing a singular official way for running c# unittests.

@van800
Copy link

van800 commented Mar 13, 2020

Normally tests are executed directly in TestRunner process. This causes any unmanaged calls to Godot api-s to fail. So if you want to test a piece of logic without calling unmanaged Godot api-s it would work. Basic example van800/SkyOfSteel@6515c3f

I have seen 2 different implementations, which overcome this limitation:

  1. TestRunner has communication channel with Application via network and tells it to load some assembly and execute some specific piece of code. http://dev-in-test.blogspot.com/2014/04/resharper-integration-tests.html
  2. TestRunner is embedded in the Application and tests are executed in the application process. For example Unity does that. It provides API for a plugin to subscribe to future test-results and run tests. Like this one https://github.com/van800/com.unity.ide.rider/blob/master/Packages/com.unity.ide.rider/Rider/Editor/UnitTesting/RiderTestRunner.cs#L67

Both approaches are not easy/fast to be implemented.

@mattrobin
Copy link

This seems like it should be a pretty important issue. Most serious projects will probably need to have unit tests, and having the standard unit testing tools available seems essential. Even for my little hobby project in Godot, I feel quite hindered by not being able to easily use standard style C# tests. It's been making me reconsider if Godot is really the engine I should be using, or if I should be switching to something that better enables regular coding practices. I can only imagine how much this deters more serious game developers from using Godot. This is probably too extreme of a blanket statement, but typically more experienced developers are more likely to rely on unit tests. And having experienced developers choose Godot as their standard tool will certainly help Godot move forward.

@martinruefenacht

This comment has been minimized.

@Calinou
Copy link
Member

Calinou commented May 22, 2020

@martinruefenacht Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.

@evan-boissonnot
Copy link

So, I'm sorry, I don't understand the release plan about this feature ?

  • How could I unit test now my godot C# project ?

Is there a way I can do it until you implement in Godot ?

  • What do you plan to do ?

Create option in IDE (in the menu) will be great
For example : stop launch and run the UI without green unit tests ?

@AlexDarigan
Copy link

AlexDarigan commented May 25, 2020

So this is self-promotion but it is also very relevant to the issue at hand.

I ran into this issue recently myself when I wanted to change from GDScript to C#. I decided to wrap WAT (The Godot GDScript Plugin) in C#. You can find the reddit thread with more detailed information and links here.

This has a GUI, CLI, Parameterized Testing & a number of other features available. Since it is built mainly using GDScript (with C# on user-interfacing scripts) it doesn't run into the same errors as the more established C# Frameworks meaning you can test any Godot Class without worry.

With that said I do think it is probably a good idea to implement some form of communication to the godot api if 1) it is relatively painless to implement 2) is generic so this fix can allow Godot to work with any standard c# test framework.

@mattrobin
Copy link

There were a couple comments that involved something to do with the UI. I just wanted to clarify that (I think) the main part of this issue requires no sort of UI changes. It's just that a test runner currently can't call any Godot related code, and for the tests to be run, the runner needs to be able to make those calls. I suspect a majority of the people who would be writing tests would be using an external IDE anyway (though I certainly may be wrong).

@Calinou Calinou changed the title Add or document a way to write unittests for Godot C# projects (e.g. with nunit3) Add or document a way to write unit tests for Godot C# projects (e.g. with nunit3) Aug 25, 2020
@van800
Copy link

van800 commented Nov 10, 2020

I managed to make a proof-of-concept integration Rider UnitTestRunner with Godot.

With the following changes:

  1. On Rider plugin side Run NUnit/XUnit tests in Godot process JetBrains/godot-support#58
  2. On the game side
    Nunit van800/godot-demo-projects#4
    Both plugins are open-source on github, anyone can try.

Still some work ahead.

  1. Make debugging work
  2. Make runner.tscn an invisible commandline, if possible.
  3. Integrate EntryPoint.cs and runner.tscn into main Godot, so users wouldn't need to copy them manually inside their project.

If I make a PR which solves (3), would it be welcomed? @neikeq
Should I create a separate proposal or we can discuss here?

image

@neikeq
Copy link

neikeq commented Nov 10, 2020

  1. Integrate EntryPoint.cs and runner.tscn into main Godot, so users wouldn't need to copy them manually inside their project.

If I make a PR which solves (3), would it be welcomed? @neikeq

Yes, that would be very welcome!

@van800
Copy link

van800 commented Nov 12, 2020

For Rider 2021.1EAP feature was merged in Rider Godot support plugin, it still requires a small preparation step - see instructions.

@Flavelius
Copy link

Flavelius commented Jan 17, 2021

@van800 It works fine for 3.2.3, but fails for 3.2.4.beta6 (using the same project), the test result is 'Inconclusive: Test not run', with GodotSharp assembly not found.

@BenMcLean

This comment has been minimized.

@Calinou
Copy link
Member

Calinou commented Feb 20, 2021

@BenMcLean Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.

@evan-boissonnot
Copy link

Hi, what's up about the prob in C# unit test.
I still get "ECall methods must be packaged into a system module." error when I want to create my own unit tests ....

@evan-boissonnot
Copy link

Have you never use Unity 3D ? There is truely way, as any C# developper can expect, to unit tests : using unit tests in VS 2019.
Not with tricks or hacks ... just using Unit Tests in IDE.

That what I need in Godot mono. :/

So for me the proposal here is not good.

@AlexDarigan
Copy link

Hi, what's up about the prob in C# unit test.
I still get "ECall methods must be packaged into a system module." error when I want to create my own unit tests ....

This is an issue about Assembly cross-talk. The JetBrains Rider plugin has a basic system working to handle this. Alternatively you can use WAT Mono which is a C# Unit Testing Plugin I made from the ground up to work with Godot without issue.

@GeorgeS2019
Copy link

Alternative based on xunit. Promising!

https://github.com/fledware/GodotXUnit
AcUjkWVqJt

@mscharley
Copy link

One option that would go a long way for me would even to just be able to unit test a library with no Godot references in it but I'm struggling to get that setup in a way that feels natural to me because of the insistence from the editor about where the sln and csproj files end up - ie, at the same level as the project.godot file. That makes it harder to have a nested testing csproj or indeed even the library project itself. The ideal project layout for me would been this but it's currently impossible as far as I can see:

/project.godot
/Project.sln
/src/Project.csproj
/lib/SharedLib.csproj
/tests/Tests.csproj

It seemed like godot didn't mind too much where the solution file was, but the csproj must be at the same level as the godot project file. Without wanting to enforce my layout on others, it would be nice if we could at least tell Godot where to look for the csproject it needs to build. The build command itself only seems to rely on the solution file but I keep getting this error (3.3.rc7.mono):

--- Debugging process stopped ---
 modules/mono/mono_gd/gd_mono_utils.cpp:370 - Microsoft.Build.Exceptions.InvalidProjectFileException: The project file could not be loaded. Could not find file "C:\[path-to-game]\Game.csproj"  C:\[path-to-game]\Game.csproj
 editor/editor_node.cpp:5497 - An EditorPlugin build callback failed.

@GeorgeS2019
Copy link

GeorgeS2019 commented Apr 13, 2021

@mscharley
Please refer to the proposal of .NET5 SDK project discussion.

Issues of existing c# project managed by Godot Mono Tools

  • GodotXUnit: for it to works with a .NET5 SDK format, we must specify exactly which c# files to be included to avoid the main Godot project csproj in .NET5 sdk format to AVOID including all cs files within the same level and below the directory which the .csproj located. This nullify the benefits brought by the simplicity of .NET5 SDK format. It is still unclear why the author of GodotXUnit mixes both the legacy .NETFramework with the .NET5 SDK formats.
  • It is still unclear how this is addressed in Godot4 mono. I am still investigating.

I do agree, we need to continue to discuss the most optimal approach for unit testing for Godot c# developers.

@MikeSchulze
Copy link

MikeSchulze commented Aug 18, 2022

@GeorgeS2019` yes there also exists a scene runner

  [TestCase(Timeout = 2000)]
        public async Task SimulateFrames()
        {
            ISceneRunner runner = ISceneRunner.Load("res://test/core/resources/scenes/TestScene.tscn", true);

            var box1 = runner.GetProperty<Godot.ColorRect>("_box1");
            // initial is white
            AssertObject(box1.Color).IsEqual(Colors.White);

            // start color cycle by invoke the function 'start_color_cycle'
            runner.Invoke("start_color_cycle");

            // we wait for 10 frames
            await runner.SimulateFrames(10);
            // after 10 frame is still white
            AssertObject(box1.Color).IsEqual(Colors.White);

            // we wait 90 more frames
            await runner.SimulateFrames(90);
            // after 100 frames the box one should be changed the color
            AssertObject(box1.Color).IsNotEqual(Colors.White);
        }

With next major release v2.3.0 the c# API will be feature complete and update the documentation

@dmlo
Copy link

dmlo commented Jan 22, 2023

I managed to make a proof-of-concept integration Rider UnitTestRunner with Godot.

With the following changes:

  1. On Rider plugin side Run NUnit/XUnit tests in Godot process JetBrains/godot-support#58
  2. On the game side
    Nunit van800/godot-demo-projects#4
    Both plugins are open-source on github, anyone can try.

Still some work ahead.

  1. Make debugging work
  2. Make runner.tscn an invisible commandline, if possible.
  3. Integrate EntryPoint.cs and runner.tscn into main Godot, so users wouldn't need to copy them manually inside their project.

If I make a PR which solves (3), would it be welcomed? @neikeq Should I create a separate proposal or we can discuss here?

image

For those coming here via google in 2023 because you want to run NUnit unit tests in Rider on Godot projects, this is the solution:

  1. Install the Godot support plugin
  2. Add the Runner.cs & Runner.tscn files from the above github repo to your Godot project in a folder called RiderTestRunner
  3. Run your tests as normal in Rider
  4. Give a star to van800 for solving your (and my) problem

Thanks @van800. This is super clean.

@a2937
Copy link

a2937 commented May 24, 2023

I managed to make a proof-of-concept integration Rider UnitTestRunner with Godot.

Is there an equivalent option for those of that use Visual Studio 2022?

@dioptryk
Copy link

I experimented a bit and was able to create a TestAdapter based on Microsoft.TestPlatform. I implemented ITestDiscoverer and ITestExecutor. When referenced in a Godot mono project (should work for both 3.x and 4.x, since it's netstandard2.0), it is properly recognized by Visual Studio and allows listing and executing tests.

Tests are listed based on special attribute in the main project assembly. The project also contains a .cs, that inherits a custom TestRunner class, that uses the same attribute to run tests. This is non-avoidable, I think, because we have to use Godot to actually execute stuff, so we need a script.

The executor simply runs Godot without window (version does not matter), but only once. It passes list of chosen tests as argument and also the Runner script as "-s" Godot parameter. Then the Runner does its thing and, for each test, outputs some special stuff on console, which is in turn understood by the executor as the specific test result.

This is a very simple solution that seems to work pretty well. No debugging, though, I'm not sure it's possible with this approach, but I've only spent two hours on this for now. Not sure if someone is interested in this.. ? I will try making a NuGet package in a few days (so there's no need for a Godot plugin), since I need a test solution myself. Also, I've no idea why, but dotnet test lists the tests correctly ("-t"), and yet when asked to execute them, tells me there are none. Probably some stupid issue somewhere.

@GeorgeS2019
Copy link

@dmitrii1991
Copy link

#432 (comment)

the version Godo 4.1 does not work. the scene starts but nothing comes bac
0
probl

@zorbathut
Copy link

@dioptryk do you have your test-framework code up anywhere? I'd love to take a look at it.

@MikeSchulze
Copy link

Just for info, I'm current working to finalize by gdUnit4 test adapter.
Features:

  • run all or a subset of tests
  • debug your tests
  • navigate to failures

Here a sneak peek
image

@dioptryk
Copy link

@zorbathut I worked a bit more on it, but I ultimately understood that Godot is not really something that should be used in unit test scenario. Try to have as many types as possible not inherit from GodotObject (implement composition in your class hierarchy) and unit test that.

I realized that what I really wanted are pipeline test results. And this is actually trivial, because you can generate test result xml in JUnit format with anything, be it GDScript or C#. You don't need to use XUnit, NUnit or whatever, you only need result files. So what I'm doing now is running godot --headless on the pipeline, starting some "scenario" scenes and verifying if they complete as expected. I also change the engine time multiplier for stuff to happen faster :) Then I am generating XML result file for my "scenarios" (see https://github.com/testmoapp/junitxml/tree/main) and publish it on Azure DevOps using PublishTestResults@2 task. And there you have it: the single test you see is actually a scene where Player character walks a few steps and picks up an Item. Then the scenario running code checks if the test succeeded (Player has an Item). If this single passes, it means that whole bunch of stuff works as expected (whole initialization, map loading, movement, inventory etc.)

obraz

@pgirald
Copy link

pgirald commented Jan 14, 2024

Hey!

Any progress on this issue? I'm currently exploring ways to run isolated tests on my scenes using xunit and the dotnet test command. Any solution that lets me test just the scene logic without the engine would be fantastic.

Alternatively, knowing how to run all the necessary Godot dependencies before executing dotnet test would work too.
The important thing is not having that "ECall methods must ...." error when loading a scene from a test method.

Currently, I'm using Godot 3.5. Cheers!

@GeorgeS2019
Copy link

GeorgeS2019 commented Feb 19, 2024

#432 (comment)

MikeSchulze/gdUnit4Net#35

using the latest gdUnit4API TestAdaptor on VS2022 from @MikeSchulze

Z0nhScGr3J

@GeorgeS2019
Copy link

GeorgeS2019 commented Mar 10, 2024

C# (Scene) Test-Driven-Development (TDD) using Godot4 in VS2022

https://youtu.be/JLI0cIYMgr0

using the latest gdUnit4API TestAdaptor from @MikeSchulze

Test.Driven.Development.in.Godot4.mp4

@van800
Copy link

van800 commented Mar 11, 2024

gdUnit4 VS integration is cool and it also turned out that it is easy to use in the JetBrains Rider.
It requires Rider 2024.1, Godot Support 2024.1.157 and enable VSTest in the Rider settings.
I have tried it on MACOSX, but I expect it to work x-plat everywhere.
There is no need to manually set the GODOT_BIN, since Rider already knows where the Godot is installed.

@a2937
Copy link

a2937 commented Aug 26, 2024

Now all we need is a stable version of GDUnit built into Godot. :P

@MikeSchulze
Copy link

Now all we need is a stable version of GDUnit built into Godot. :P

OMG, this requires a complete rewrite in c++ :)
Maybe binding by the GdExtension, but this is also a lot of work to do.

@a2937
Copy link

a2937 commented Sep 2, 2024

Can't they like publicize some of the existing test runner stuff? They already have pretty good tests for the GDScript implementation itself. https://github.com/godotengine/godot/tree/master/modules/gdscript/tests

@AThousandShips
Copy link
Member

What do you mean publicize?

@a2937
Copy link

a2937 commented Sep 2, 2024

Make it usable outside of the source code for the Godot Game engine.

@AThousandShips
Copy link
Member

Gotcha, never heard that term used like that (was confused as the code is public)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests