API tooling goes amok in Jenkins, need some extra logging #782

iloveeclipse · 2023-10-05T14:52:25Z

Something similar to #771 (but not on Java 21), reported for this change (which doesn't break or add or remove any API): eclipse-platform/eclipse.platform.ui#1097.
Full maven log: https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-1097/4/consoleFull

Relevant part says that most of JFace API was removed. !?!?

I can only assume that something completely weird happening in the API application, like not finding any classes or comparing "empty" state with baseline etc.

Since we can't find the root cause, we should at least provide more data while reporting API errors.

This could be:

exact Java version used,
some baseline description (path/content or something like that)
whatever else that could be of interest from API tooling point of view

15:12:24.531 [INFO] --- tycho-p2-extras:4.0.3:compare-version-with-baselines (compare-attached-artifacts-with-release) @ org.eclipse.jface ---
15:12:24.539 [INFO] 
15:12:24.539 [INFO] --- tycho-apitools:4.0.3:verify (verify) @ org.eclipse.jface ---
15:12:24.541 [INFO] Resolve API baseline for org.eclipse.platform:org.eclipse.jface:eclipse-plugin:3.32.0-SNAPSHOT
15:12:25.211 [INFO] Resolve API tools runtime from MavenRepositoryLocation [location=https://download.eclipse.org/eclipse/updates/4.30-I-builds/]...
383 API ERRORS
[API ERROR] File MANIFEST.MF at line 5: The major version should be incremented in version 3.32.0, since API breakage occurred since version 3.31.0 - The type org.eclipse.jface.action.IMenuListener2 in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.dialogs.MessageDialog in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.viewers.AbstractListViewer in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.action.StatusLineManager in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.viewers.TreeSelection in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.bindings.keys.SWTKeyLookup in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.dialogs.IconAndMessageDialog in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.viewers.TableViewerColumn in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.action.LegacyActionTools in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.viewers.TableViewerFocusCellManager in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.bindings.ISchemeListener in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.action.StatusLineLayoutData in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.bindings.BindingManager in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.dialogs.IPageChangedListener in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.preference.IntegerFieldEditor in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.viewers.TreeColumnViewerLabelProvider in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.util.Throttler in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.bindings.Binding in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.dialogs.IMessageProvider in org.eclipse.jface_3.32.0 is no longer an API
- The type org.eclipse.jface.wizard.Wizard in org.eclipse.jface_3.32.0 is no longer an API
362 more change(s)... (location: /home/jenkins/agent/workspace/eclipse.platform.ui_PR-1097/bundles/org.eclipse.jface/META-INF/MANIFEST.MF)

The text was updated successfully, but these errors were encountered:

vik-chand · 2023-10-05T17:13:02Z

Full maven log: https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-1097/4/consoleFull
Unable to open it.

Irrespective of that, extra logging would be helpful especially when this is reported in the log ( instead of workspace - where user can actually see the baseline and Java version)

laeubi · 2023-10-06T15:59:43Z

API tooling goes amok in Jenkins

I think one way would be to run this as a local build when it happens and then one can enable debug output of maven to see more details of what Tycho has computed...

laeubi · 2023-10-07T15:57:24Z

Tycho SNAPSHOTS will soon contain an improved ApiToolsMojo:

Call the API tools directly without using ApiAnalysisApplication eclipse-tycho/tycho#2899

this will allow to set a debug flag to print out more details about the used API baseline including EE and ApiTools version:

<plugin>
	<groupId>org.eclipse.tycho</groupId>
	<artifactId>tycho-apitools-plugin</artifactId>
	<configuration>
		<debug>false</debug>
	</configuration>
</plugin>

laeubi · 2023-10-12T09:24:24Z

@iloveeclipse I have enabled Tycho 4.0.4-SNAPSHOT now:

Use Tycho 4.0.4-SNAPSHOT eclipse-platform/eclipse.platform.releng.aggregator#1437

I consider this issue as done with that (as it is not a pure PDE issue actually), so if you still see any issues please let me know.

stephan-herrmann · 2023-10-12T22:09:08Z

Would the below errors from https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-1447/5 fit the bill of this issue?

23:37:08  [ERROR] Failed to execute goal org.eclipse.tycho:tycho-apitools-plugin:4.0.4-SNAPSHOT:verify (verify) on project org.eclipse.jdt.core.compiler.batch: There are API errors:
23:37:08  [ERROR] src/org/eclipse/jdt/core/compiler/CategorizedProblem.java:63 The type org.eclipse.jdt.core.compiler.CategorizedProblem has been removed from org.eclipse.jdt.core.compiler.batch_3.36.0
23:37:08  [ERROR] src/org/eclipse/jdt/core/compiler/CharOperation.java:29 The type org.eclipse.jdt.core.compiler.CharOperation has been removed from org.eclipse.jdt.core.compiler.batch_3.36.0
23:37:08  [ERROR] src/org/eclipse/jdt/core/compiler/CompilationProgress.java:29 The type org.eclipse.jdt.core.compiler.CompilationProgress has been removed from org.eclipse.jdt.core.compiler.batch_3.36.0
23:37:08  [ERROR] src/org/eclipse/jdt/core/compiler/IProblem.java:256 The type org.eclipse.jdt.core.compiler.IProblem has been removed from org.eclipse.jdt.core.compiler.batch_3.36.0
23:37:08  [ERROR] src/org/eclipse/jdt/core/compiler/InvalidInputException.java:22 The type org.eclipse.jdt.core.compiler.InvalidInputException has been removed from org.eclipse.jdt.core.compiler.batch_3.36.0
23:37:08  [ERROR] src/org/eclipse/jdt/core/compiler/batch/BatchCompiler.java:29 The type org.eclipse.jdt.core.compiler.batch.BatchCompiler has been removed from org.eclipse.jdt.core.compiler.batch_3.36.0
23:37:08  [ERROR] META-INF/MANIFEST.MF:6 The major version should be incremented in version 3.36.0, since API breakage occurred since version 3.35.0

Read: Analysis points to an existing type and complains that it has been removed. Really?

iloveeclipse · 2023-10-13T04:33:57Z

Same here: https://ci.eclipse.org/jdt/job/eclipse.jdt.debug-github/job/PR-324/1/console
Lot of errors for removed types on a trivial, unrelated PR.

@laeubi : please fix tycho or revert the changes to use latest tycho.

laeubi · 2023-10-13T05:13:51Z

@stephan-herrmann I don't think that these sarcastic undertones help to resolve the issue, I just want to remind to this (still not resolve) JDT issue where jdt is unable to compile valid java code into valid byte code (really? ;-) what works fine on javac (and now with Java 21 third preview will reach even greater audience):

Import static with protected base class yiels wrong class reference compiled in resulting classfile eclipse-jdt/eclipse.jdt.core#809

and yes there are some "insane" sideeffects possible with API tools see for example:

API tools gets mad when changing extension point and complains with strange error but only if implemented #378

and no one really likes to support bugfixing and/or improvements (I think @vik-chand is the only one somehow dedicated to it) in that area, why I suggested to discontinue this:

https://www.eclipse.org/lists/pde-dev/msg02945.html

but it does not get much attention back there why we have to live with that for a while.

So here are some instructions for anyone who is willing to help understand/fixing the problem:

go to the IDE and checkout the PR that reproduces the problem with Team > Fetch Gihub Pull Request, make sure you select the eclipse repo as a source)
Open a terminal and go to the folder of the failing project
Make sure latest Maven 3.9 is available
run mvn clean install -Papi-check -DskipTests -Dtycho.apitools.debug

If you can see the problem (you should already get some debug output), go back to your IDE:

Create a new remote debugging session for project org.eclipse.pde.api.tools
put a breakpoint at BaseApiAnalyzer.analyzeComponent(BuildState, IApiFilterStore, Properties, IApiBaseline, IApiComponent, IBuildContext, IProgressMonitor)
happy debugging!

Would the below errors from https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-1447/5 fit the bill of this issue

This one builds fine with the given instructions above, could you please try if you can reproduce the problem or any special jdt specific settings are required to reproduce the problem?

Same here: https://ci.eclipse.org/jdt/job/eclipse.jdt.debug-github/job/PR-324/1/console

The PR fails to compile for me locally:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:4.0.4-SNAPSHOT:compile (default-compile) on project org.eclipse.jdt.debug.tests: Compilation failure: Compilation failure: 
[ERROR] org.eclipse.jdt.debug.tests/tests/org/eclipse/jdt/debug/tests/core/ClasspathContainerTests.java:[262] 
[ERROR] 	JavaIndexer.generateIndexForJar(libraryFile.getAbsolutePath(), indexPath.toOSString());

laeubi · 2023-10-13T05:17:56Z

@stephan-herrmann @iloveeclipse can you please "replay" a failing build and add -Dtycho.apitools.debug to the Jenkins file?

for example (please adjust to jdt accordingly):

iloveeclipse · 2023-10-13T05:30:38Z

can you please "replay" a failing build and add -Dtycho.apitools.debug to the Jenkins file?

See https://ci.eclipse.org/jdt/job/eclipse.jdt.debug-github/job/PR-324/2/

iloveeclipse · 2023-10-13T05:32:46Z

can you please "replay" a failing build and add -Dtycho.apitools.debug to the Jenkins file?

and https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-1447/7/

iloveeclipse · 2023-10-13T05:37:57Z

The PR fails to compile for me locally:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:4.0.4-SNAPSHOT:compile (default-compile) on project org.eclipse.jdt.debug.tests: Compilation failure: Compilation failure: 
[ERROR] org.eclipse.jdt.debug.tests/tests/org/eclipse/jdt/debug/tests/core/ClasspathContainerTests.java:[262] 
[ERROR] 	JavaIndexer.generateIndexForJar(libraryFile.getAbsolutePath(), indexPath.toOSString());

Which exact error was it? One line is missing. Note, PR eclipse-jdt/eclipse.jdt.debug#324 doesn't change any signatures, and only touches PListParser in a different bundle. So whatever compilation error happened in ClasspathContainerTests it seem to be unrelated to the PR in question. Also I see no errors in the IDE with that PR.

laeubi · 2023-10-13T05:43:22Z

I found another build and replayed it but now now API tools errors:

https://ci.eclipse.org/equinox/job/equinox/job/PR-337/

so it seems to be some kind of race condition in API tools or it depend on what is available on the I-Build repository...

merks · 2023-10-13T05:51:32Z

FYI, it's probably not helpful, but I see this kind of thing happen in my SDK IDE occasionally too. Sometimes there are just a sea of errors about removed things that are obviously not removed. The only reliable way I found to "fix" this is to delete the API baseline in the preferences and to let the Oomph setup build a new one.

laeubi · 2023-10-13T06:20:41Z

Which exact error was it?

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:4.0.4-SNAPSHOT:compile (default-compile) on project org.eclipse.jdt.debug.tests: Compilation failure: Compilation failure: 
[ERROR] eclipse.jdt.debug/org.eclipse.jdt.debug.tests/tests/org/eclipse/jdt/debug/tests/core/ClasspathContainerTests.java:[262] 
[ERROR] 	JavaIndexer.generateIndexForJar(libraryFile.getAbsolutePath(), indexPath.toOSString());
[ERROR] 	            ^^^^^^^^^^^^^^^^^^^
[ERROR] The method generateIndexForJar(String, String) is undefined for the type JavaIndexer
[ERROR] 1 problem (1 error)
[ERROR] -> [Help 1]

Also I see no errors in the IDE with that PR.

I just used the maven comandline for compile so maybe this is an intermediate problem.

FYI, it's probably not helpful, but I see this kind of thing happen in my SDK IDE occasionally too

At least it helps to understand that this is nothing that can be fixed at Tycho side. If it occurs again, it might be good to put a breakpoint at where the error is generated to see what state ApiTools make think the is a problem.

iloveeclipse · 2023-10-13T06:21:04Z

Debug info & API errors:

laeubi · 2023-10-13T06:23:42Z

The debug says for the simple (compiler) case:

07:35:42  API Tools version: 1.3.200.v20231008-1448
07:35:44  Project component was found
07:35:44  ==== org.eclipse.jdt.core.compiler.batch - baseline ====
07:35:44  org.eclipse.jdt.core.compiler.batch 3.35.0 -- org.eclipse.jdt.core.compiler.batch-3.35.0.v20230814-2020.jar
07:35:44  System Component:
07:35:44  	Version: JavaSE-21
07:35:44  	Location: /opt/tools/java/openjdk/jdk-21/21+35-2513
07:35:44  	Execution Environment: JavaSE-21

anything suspicious?

Here is the output from more complex failure with a larger baseline:

07:32:28  API Tools version: 1.3.200.v20231008-1448
07:32:30  Project component was found
07:32:30  ==== org.eclipse.jdt.launching - baseline ====
07:32:30  org.eclipse.core.commands 3.11.100 -- org.eclipse.core.commands-3.11.100.v20230708-0916.jar
07:32:30  org.eclipse.equinox.common 3.18.100 -- org.eclipse.equinox.common-3.18.100.v20230730-1117.jar
07:32:30  org.eclipse.core.contenttype 3.9.100 -- org.eclipse.core.contenttype-3.9.100.v20230630-1232.jar
07:32:30  org.eclipse.equinox.preferences 3.10.300 -- org.eclipse.equinox.preferences-3.10.300.v20230630-1303.jar
07:32:30  org.eclipse.equinox.registry 3.11.300 -- org.eclipse.equinox.registry-3.11.300.v20230801-1826.jar
07:32:30  org.eclipse.core.expressions 3.9.100 -- org.eclipse.core.expressions-3.9.100.v20230726-0617.jar
07:32:30  org.eclipse.core.runtime 3.29.0 -- org.eclipse.core.runtime-3.29.0.v20230726-0617.jar
07:32:30  org.eclipse.core.filesystem 1.10.100 -- org.eclipse.core.filesystem-1.10.100.v20230717-1727.jar
07:32:30  org.eclipse.core.jobs 3.15.0 -- org.eclipse.core.jobs-3.15.0.v20230808-1403.jar
07:32:30  org.eclipse.core.resources 3.19.100 -- org.eclipse.core.resources-3.19.100.v20230814-1312.jar
07:32:30  org.eclipse.equinox.app 1.6.300 -- org.eclipse.equinox.app-1.6.300.v20230630-1303.jar
07:32:30  org.eclipse.core.variables 3.6.100 -- org.eclipse.core.variables-3.6.100.v20230726-0617.jar
07:32:30  org.eclipse.debug.core 3.21.100 -- org.eclipse.debug.core-3.21.100.v20230731-1425.jar
07:32:30  org.osgi.service.prefs 1.1.2 -- org.osgi.service.prefs-1.1.2.202109301733.jar
07:32:30  org.eclipse.jdt.core 3.35.0 -- org.eclipse.jdt.core-3.35.0.v20230814-1356.jar
07:32:30  org.eclipse.text 3.13.100 -- org.eclipse.text-3.13.100.v20230801-1334.jar
07:32:30  org.eclipse.jdt.core.compiler.batch 3.35.0 -- org.eclipse.jdt.core.compiler.batch-3.35.0.v20230814-2020.jar
07:32:30  org.eclipse.jdt.debug 3.21.100 -- org.eclipse.jdt.debug-3.21.100.v20230728-0612.jar
07:32:30  org.eclipse.jdt.launching 3.20.100 -- org.eclipse.jdt.launching-3.20.100.v20230814-1810.jar
07:32:30  System Component:
07:32:30  	Version: JavaSE-17
07:32:30  	Location: /opt/tools/java/temurin/jdk-17/jdk-17.0.6+10
07:32:30  	Execution Environment: JavaSE-17

akurtakov · 2023-10-13T06:31:10Z

It might look like hijacking the thread but we should seriously ask ourselves - Is anyone willing to spend the time supporting PDE API tools?
If no one steps up https://github.com/eclipse-tycho/tycho/blob/master/RELEASE_NOTES.md#new-tycho-baseline-plugin is worth considering as alternative. https://bndtools.org/ provides baselining in the IDE.

laeubi · 2023-10-13T06:52:25Z

It might look like hijacking the thread but we should seriously ask ourselves - Is anyone willing to spend the time supporting PDE API tools?

Last time only @vik-chand stepped up, I tried to fix some findings while developing Tycho but this seems hard to track as it is not always reproducible.

https://bndtools.org/ provides baselining in the IDE.

m2e also support calling the Tycho-Mojo for validation.

laeubi · 2023-10-13T07:12:53Z

I now tried to reproduce the problem locally with even Java 21 enabled, but the build succeeds as expected (except some warnings), so it does not seem to be directly related to Tycho.

I even tried with a complete empty m2 repo to make sure there are no leftovers from previous builds. The only thing i noticed is that it sometime uses Java 17 and sometimes Java 21 as system component, this might be the autodiscovery feature of JDT, so if one can point me to how to disable this I can probably set that flag at Tycho (even though it does not seem to make api tools fail) maybe this is more some jobs running in parallel? I'll later try to install a job listener so verify this.

vik-chand · 2023-10-13T07:38:13Z

It might look like hijacking the thread but we should seriously ask ourselves - Is anyone willing to spend the time supporting PDE API tools?

Last time only @vik-chand stepped up, I tried to fix some findings while developing Tycho but this seems hard to track as it is not always reproducible.

https://bndtools.org/ provides baselining in the IDE.

m2e also support calling the Tycho-Mojo for validation.

If this recreates in the IDE, I could fix it. I tried to recreate it locally but it seemed to work.

vik-chand · 2023-10-13T07:41:03Z

FYI, it's probably not helpful, but I see this kind of thing happen in my SDK IDE occasionally too. Sometimes there are just a sea of errors about removed things that are obviously not removed. The only reliable way I found to "fix" this is to delete the API baseline in the preferences and to let the Oomph setup build a new one.

This is a good hint. For some reason, the API baseline has incorrect system library ( may be also incorrect plugins)

vik-chand · 2023-10-13T07:44:08Z

I found another build and replayed it but now now API tools errors:
* https://ci.eclipse.org/equinox/job/equinox/job/PR-337/
so it seems to be some kind of race condition in API tools or it depend on what is available on the I-Build repository...

Could it because of parallel API analysis @iloveeclipse ? Not sure if we disabled this during our API tool test cases.

vogella · 2023-10-13T07:54:56Z

It might look like hijacking the thread but we should seriously ask ourselves - Is anyone willing to spend the time supporting PDE API tools?

Last time only @vik-chand stepped up, I tried to fix some findings while developing Tycho but this seems hard to track as it is not always reproducible.

https://bndtools.org/ provides baselining in the IDE.

m2e also support calling the Tycho-Mojo for validation.

I always assumed that the API tooling for Tycho re-uses the API tooling from PDE. If that is not the case, I suggest to deprecate the PDE API tooling and mark it for deletion so that we COULD (do not have to) disable it after the API deletion period.

akurtakov · 2023-10-13T07:59:38Z

I always assumed that the API tooling for Tycho re-uses the API tooling from PDE. If that is not the case, I suggest to deprecate the PDE API tooling and mark it for deletion so that we COULD (do not have to) disable it after the API deletion period.

Tycho has 2 different baseline verifications plugins - one on top of PDE API tools (used in Platform builds) and another based on bnd .

Should help to understand strange API analysis errors. See eclipse-pde/eclipse.pde#782

iloveeclipse · 2023-10-13T13:19:32Z

So, after setting two VM properties on jenkins still API errors...
See https://ci.eclipse.org/jdt/job/eclipse.jdt.debug-github/job/PR-324/3/consoleFull

Related log part:
api_errors_launching_bundle.log

iloveeclipse · 2023-10-13T13:31:00Z

An observation: we see an increased number of API failures on jenkins from various repositories after switching to tycho 4.0.4 in eclipse-platform/eclipse.platform.releng.aggregator#1437.

If API analysis was "just" not stable before, and mostly on JDT/Java 21, now it seem to be completely broken, independently on the repository.

Can we revert to tycho 4.0.3 or at least check what could be changed in 4.0.4 that causes such major increase of API errors?

iloveeclipse · 2023-10-13T14:34:47Z

I see the tycho doesn't use ApiAnalysisApplication now, since eclipse-tycho/tycho#2899

I see that the project is imported into the workspace:
https://github.com/eclipse-tycho/tycho/blob/16eba006183677d3ee5db10d0da839f905fcec18/tycho-apitools-plugin/src/main/java/org/eclipse/tycho/apitools/ApiAnalysis.java#L141

and the workspace autobuild is disabled:
https://github.com/eclipse-tycho/tycho/blob/16eba006183677d3ee5db10d0da839f905fcec18/tycho-apitools-plugin/src/main/java/org/eclipse/tycho/apitools/ApiAnalysis.java#L98

What I don't see is: where do we build the project, wait till it is built & check that compilation had no errors?
Just looking at the "type XYZ is missing" I simply assume the API builder runs over not fully compiled project, or project that had compilation errors.

laeubi · 2023-10-13T16:58:37Z

If API analysis was "just" not stable before, and mostly on JDT/Java 21, now it seem to be completely broken, independently on the repository

If it is reproducible on a PR please let me know, currently the problem seems only on Jenkins infra what makes it hard to tackle.

What I don't see is: where do we build the project, wait till it is built & check that compilation had no errors?

The project is not compiled because API tools is using the JDT-AST to analyses projects that are of source type, if it is not a source type the classes compiled by Tycho are used, or we use the jar from the target platform.

iloveeclipse · 2023-10-13T17:41:56Z

This PR constantly produces API errors in Jenkins: eclipse-jdt/eclipse.jdt.debug#324
Note, Jenkins has now enabled debug flag for API tooling in this repo.

API tools is using the JDT-AST to analyses projects that are of source type

This surprises me a lot. But if API tooling goes over sources, how it can miss whole types?
Could you point me to the place where API tooling traverses the source tree? So far I assumed there is a "usual" project builder that is invoked by the build manager...

Could you also in tycho add error log listener to Platform and report all errors/warnings in the platform to the maven output?

I wonder if the AST couldn't be created because something couldn't be resolved but we miss these errors.

iloveeclipse · 2023-10-13T17:49:50Z

Thinking more, I wonder if the build is needed because AST creation needs some referenced classes to be generated / available to resolve something. Could you please trigger project build in your tycho code, in the same way it was done in the original code of ApiAnalysisApplication?

laeubi · 2023-10-13T18:16:20Z

This PR constantly produces API errors in Jenkins: eclipse-jdt/eclipse.jdt.debug#324

I checked out the pr and run

mvn clean verify --batch-mode --fail-at-end -Dmaven.repo.local=/tmp/.m2/repository -Pbuild-individual-bundles -Ptest-on-javase-19 -Pbree-libs -Papi-check -Dcompare-version-with-baselines.skip=false -Dproject.build.sourceEncoding=UTF-8 -DDetectVMInstallationsJob.disabled=true -Dtycho.apitools.debug -DtrimStackTrace=false

but build is successful :-\

So there must be something in Jenkins that is different and makes it behave not like a local (clean) build....

This surprises me a lot. But if API tooling goes over sources, how it can miss whole types?

I have no clue ... especially I would expect it to always fail or always succeed ...

Could you point me to the place where API tooling traverses the source tree?

The main entry point (that is used by Tycho) is:

eclipse.pde/apitools/org.eclipse.pde.api.tools/src/org/eclipse/pde/api/tools/internal/model/ApiModelFactory.java

Lines 109 to 130 in 27cf508

    
           public static IApiComponent newApiComponent(IApiBaseline baseline, IPluginModelBase model) throws CoreException { 
        
           	BundleDescription bundleDescription = model.getBundleDescription(); 
        
           	if (bundleDescription == null) { 
        
           		return null; 
        
           	} 
        
           	String location = bundleDescription.getLocation(); 
        
           	if (location == null) { 
        
           		return null; 
        
           	} 
        
           	BundleComponent component; 
        
           	IResource resource = model.getUnderlyingResource(); 
        
           	IProject project = resource != null ? resource.getProject() : null; 
        
           	if (project != null && project.exists() && !Util.isBinaryProject(project)) { 
        
           		component = new ProjectComponent(baseline, location, model, getBundleID()); 
        
           	} else { 
        
           		component = new BundleComponent(baseline, location, getBundleID()); 
        
           	} 
        
           	if (component.isValidBundle()) { 
        
           		return component; 
        
           	} 
        
           	return null; 
        
           }

as you can see there is a distinction between a "binary" BundleComponent what works on precompiled jar but has no support for getting source references see: #785 what would be much better because actually always have a binary compiled jar what we actually want to compare ...

Second case is the ProjectComponent where we have a (java) project and it (as far as I can understand) extract things from the AST/JDT Model to be able to give source locations for error markers and for javadoc @since tags:

https://github.com/eclipse-pde/eclipse.pde/blob/master/apitools/org.eclipse.pde.api.tools/src/org/eclipse/pde/api/tools/internal/model/ProjectComponent.java

As explained here #782 (comment) one can even (remote) debug what happens exactly there when running with mvnDebug, but problem for me is that

I'm only "beginner" at API tools internals and not expert on JDT so some of my understanding/assumption might be wrong
if I build such failing build locally it does not fail even if I run it multiple times, it simply works as expected so it is currently a bit guess work.

Could you please trigger project build in your tycho code, in the same way it was done in the original code of ApiAnalysisApplication?

Actually compiling the code (again) is what should be avoided as API this often has lead to (other) strange errors where there was an (intermediate) compilation problem. Also the old way showed exactly these errors we see now, so I first like to understand the cause of it and and make it more deterministic. I hope the added debugging might give a hint on whats going on when it fails.

laeubi · 2023-10-13T18:22:09Z

Thinking more, I wonder if the build is needed because AST creation needs some referenced classes to be generated / available to resolve something

I'll try tomorrow with a fresh checkout to make sure there is nothing from a IDE build there...

iloveeclipse · 2023-10-13T18:25:46Z

I wonder if we must explicitly wait for the JDT indexer.
I haven't worked with AST in JDT but I guess it might rely on information from Indexer, and if that one is not done yet, we might "miss" types. Note, jenkins instances on Eclipse org are pretty limited, so if they have only two cores, indexer job might simply have no chance to complete, while locally it might use much more cores (even my old notebook has 8 cores, and 16 on the workstation). This would explain why the fails are unstable.

iloveeclipse · 2023-10-13T18:30:22Z

Even worse it seems JDT has its own JobManager org.eclipse.jdt.internal.core.search.processing.JobManager, so IndexManager is still running (but this hopefully will not harm).

This is the one we might need to wait for.

iloveeclipse · 2023-10-13T19:49:28Z

JavaModelManager.getIndexManager().waitForIndex(true, null);

laeubi · 2023-10-14T03:58:35Z

I'll try tomorrow with a fresh checkout to make sure there is nothing from a IDE build there...

This did the trick for me to reproduce the problem on local machine! So it seems that the IDE is generating some kind of file(s) that later makes the API tools succeed.

So I head over to my previously working checkout and deleted the bin/ folder and now it is also failing there... so for some reason either JDT or API tools still requires the compiled classfiles (but not always), even though if I just remove all builders for example from the batch compiler, delete the bin folder and then open it in a new workspace I don't get any immediate errors/problem.

If anyone has a clue how to reproduce this (missing classfile results in API error), please provide an integration-test to demonstrate the issue, it seems the current used one here is not able to show the problem:
https://github.com/eclipse-tycho/tycho/blob/master/tycho-its/src/test/java/org/eclipse/tycho/test/apitools/ApiToolsTest.java it seems I can actually reproduce it there as well... I just misread what API tools has complained about :-\

laeubi · 2023-10-14T04:46:32Z

so for some reason either JDT or API tools still requires the compiled classfiles (but not always)

It might be related to org.eclipse.pde.api.tools.internal.model.ProjectTypeContainer.getPackageNames() what (for unknown reasons) do not ask JDT for package names but traverses the output directory (see also #791) to collect package names. There might be other places in API tools as well that do similar so maybe not even the classfiles but the folder names are the missing piece here...

laeubi · 2023-10-14T04:59:09Z

I now have prepared a fix for Tycho here:

Set the output folder of the imported project to the maven location eclipse-tycho/tycho#2922

also created:

[apitools] Use the AST/JDT to discover package names #801

@vik-chand would be great if you can take a look at:
https://github.com/eclipse-pde/eclipse.pde/issues?q=is%3Aopen+is%3Aissue+label%3Aapi-tools

laeubi · 2023-10-14T12:06:15Z

@iloveeclipse @stephan-herrmann the bugfix is online now, please restart the failing builds to verify if it now works without problems.

stephan-herrmann · 2023-10-14T13:51:41Z

@iloveeclipse @stephan-herrmann the bugfix is online now, please restart the failing builds to verify if it now works without problems.

Build jdt.core/PR-1447/8 triggered by @iloveeclipse (thanks!) was successful.

merks · 2023-10-14T13:58:49Z

@iloveeclipse @laeubi

Very impressive detective work and teamwork! 🥇

vik-chand · 2023-10-16T12:59:12Z

@vik-chand would be great if you can take a look at: https://github.com/eclipse-pde/eclipse.pde/issues?q=is%3Aopen+is%3Aissue+label%3Aapi-tools

I am on vacation this week. Will look at this list next week

laeubi · 2023-10-28T16:17:21Z

JavaModelManager.getIndexManager().waitForIndex(true, null);

Is there a way to disable indexing all together? This seem to use additional memory and I like to see if it will work without:

JRTUtil.getJrtSystem(...) permanently consumes a large amount of memory eclipse-jdt/eclipse.jdt.core#1526 (comment)

That should reduce possible side effects on API analysis. See eclipse-pde/eclipse.pde#782

Should help to understand strange API analysis errors. See eclipse-pde/eclipse.pde#782

iloveeclipse added the api-tools label Oct 5, 2023

laeubi assigned vik-chand Oct 5, 2023

laeubi mentioned this issue Oct 7, 2023

[apitools] BaseApiAnalyzer should have a way to discover the source without a java project #785

Open

laeubi closed this as completed Oct 12, 2023

iloveeclipse reopened this Oct 13, 2023

iloveeclipse mentioned this issue Oct 13, 2023

Apply project settings to o.e.equinox.security.UI eclipse-equinox/equinox#337

Merged

iloveeclipse added a commit to eclipse-jdt/eclipse.jdt.core that referenced this issue Oct 13, 2023

Enable debug for API tooling on Jenkins

eddafe9

Should help to understand strange API analysis errors. See eclipse-pde/eclipse.pde#782

HeikoKlare mentioned this issue Oct 13, 2023

Fix test for save state cleanup on snapshot save / bug 297635 #460 eclipse-platform/eclipse.platform#737

Merged

gireeshpunathil mentioned this issue Oct 13, 2023

Toggle OSGi and Eclipse plugin creation options appropriately #800

Closed

iloveeclipse mentioned this issue Oct 13, 2023

Resolve PLIST entities to zero-length content. eclipse-jdt/eclipse.jdt.debug#324

Merged

HeikoKlare mentioned this issue Oct 13, 2023

Update the JFace.action tests to Junit4 eclipse-platform/eclipse.platform.ui#1220

Merged

This was referenced Oct 14, 2023

Set the output folder of the imported project to the maven location eclipse-tycho/tycho#2922

Merged

[apitools] Use the AST/JDT to discover package names #801

Open

laeubi closed this as completed Oct 14, 2023

iloveeclipse mentioned this issue Nov 6, 2023

PDE Jenkins build broken, API tools report errors #888

Closed

robstryker pushed a commit to robstryker/eclipse.jdt.core that referenced this issue Jul 18, 2024

Disable DetectVMInstallationsJob in Jenkins

0b08ed7

That should reduce possible side effects on API analysis. See eclipse-pde/eclipse.pde#782

robstryker pushed a commit to robstryker/eclipse.jdt.core that referenced this issue Jul 18, 2024

Enable debug for API tooling on Jenkins

afd370b

Should help to understand strange API analysis errors. See eclipse-pde/eclipse.pde#782

API tooling goes amok in Jenkins, need some extra logging #782

API tooling goes amok in Jenkins, need some extra logging #782

Comments

iloveeclipse commented Oct 5, 2023

vik-chand commented Oct 5, 2023

laeubi commented Oct 6, 2023

laeubi commented Oct 7, 2023

laeubi commented Oct 12, 2023

stephan-herrmann commented Oct 12, 2023

iloveeclipse commented Oct 13, 2023

laeubi commented Oct 13, 2023

So here are some instructions for anyone who is willing to help understand/fixing the problem:

laeubi commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

laeubi commented Oct 13, 2023

merks commented Oct 13, 2023

laeubi commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

laeubi commented Oct 13, 2023 • edited Loading

akurtakov commented Oct 13, 2023

laeubi commented Oct 13, 2023

laeubi commented Oct 13, 2023

vik-chand commented Oct 13, 2023

vik-chand commented Oct 13, 2023

vik-chand commented Oct 13, 2023

vogella commented Oct 13, 2023

akurtakov commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

laeubi commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

laeubi commented Oct 13, 2023

laeubi commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

iloveeclipse commented Oct 13, 2023

laeubi commented Oct 14, 2023 • edited Loading

laeubi commented Oct 14, 2023

laeubi commented Oct 14, 2023

laeubi commented Oct 14, 2023

stephan-herrmann commented Oct 14, 2023

merks commented Oct 14, 2023

vik-chand commented Oct 16, 2023

laeubi commented Oct 28, 2023

laeubi commented Oct 13, 2023 •

edited

Loading

laeubi commented Oct 14, 2023 •

edited

Loading