Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot compile queries using nuget package Microsoft.CodeAnalysis 2.0.0 #13

Open
wangzq opened this issue Mar 26, 2017 · 8 comments
Open
Labels

Comments

@wangzq
Copy link

wangzq commented Mar 26, 2017

To repro, create a new query and add nuget package Microsoft.CodeAnalysis which is 2.0.0, add following code:

var workspace = new AdhocWorkspace();

Add missing namespace imports and ensure it works in Linqpad.

Now try to generate an exe with lpless, it will fail to compile since it cannot correctly add the assemblies from the nuget package.

@atifaziz atifaziz added the bug label Mar 27, 2017
@atifaziz
Copy link
Owner

atifaziz commented Mar 27, 2017

I prepared a LINQPad query as you described and named it LinqPadless-Issue-13.linq. Then I compiled it as follows:

lpless -v I:\LINQPad\Queries\LinqPadless-Issue-13.linq

The output from the compilation was:

I:\LINQPad\Queries\LinqPadless-Issue-13.linq
  <Query Kind="Expression">
    <NuGetReference>Microsoft.CodeAnalysis</NuGetReference>
  </Query>
  Packages referenced (1):
    Microsoft.CodeAnalysis
  Packages directory: I:\LINQPad\Queries\packages
  Packages target: .NETFramework,Version=v4.5.2
  Querying latest version of Microsoft.CodeAnalysis
    [INF]   GET https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json
    [INF]   GET https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json
    [INF]   GET https://www.myget.org/F/raboof/api/v2/FindPackagesById()?id='Microsoft.CodeAnalysis'
    [INF]   OK https://www.myget.org/F/raboof/api/v2/FindPackagesById()?id='Microsoft.CodeAnalysis' 65ms
    [INF]   OK https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json 636ms
    [INF]   OK https://api.nuget.org/v3/registration1-gz/microsoft.codeanalysis/index.json 639ms
    Using version 2.0.0
  Resolving references...
    Microsoft.CodeAnalysis 2.0.0
      Microsoft.CodeAnalysis.CSharp.Workspaces 2.0.0
        Microsoft.CodeAnalysis.CSharp 2.0.0
          Microsoft.CodeAnalysis.Common 2.0.0
        Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
      Microsoft.CodeAnalysis.VisualBasic.Workspaces 2.0.0
        Microsoft.CodeAnalysis.VisualBasic 2.0.0
          Microsoft.CodeAnalysis.Common 2.0.0
        Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
  Warning! Packages with no references for .NETFramework,Version=v4.5.2:
    Microsoft.CodeAnalysis.CSharp.Workspaces 2.0.0
    Microsoft.CodeAnalysis.CSharp 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
    Microsoft.CodeAnalysis.VisualBasic.Workspaces 2.0.0
    Microsoft.CodeAnalysis.VisualBasic 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
  "C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe" I:\LINQPad\Queries\LinqPadless-Issue-13.cs /r:System.dll /r:Microsoft.CSharp.dll /r:System.Core.dll /r:System.Data.dll /r:System.Data.Entity.dll /r:System.Transactions.dll /r:System.Xml.dll /r:System.Xml.Linq.dll /r:System.Data.Linq.dll /r:System.Drawing.dll /r:System.Data.DataSetExtensions.dll
    Microsoft (R) Visual C# Compiler version 1.3.1.60616
    Copyright (C) Microsoft Corporation. All rights reserved.
    
    LinqPadless-Issue-13.cs(23,15): error CS0234: The type or namespace name 'CodeAnalysis' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)
C# compiler finished with a non-zero exit code of 1.
lpless.exe Error: 0 : System.Exception: C# compiler finished with a non-zero exit code of 1.
   at LinqPadless.Program.GenerateExecutable(String cscPath, String queryFilePath, String packagesPath, QueryLanguage queryKind, String source, IEnumerable`1 imports, IEnumerable`1 references, IndentingLineWriter writer) in A:\LinqPadless\src\Program.cs:line 673
   at LinqPadless.Program.<>c__DisplayClass16_0.<GenerateExecutable>b__0(String queryFilePath, String packagesPath, QueryLanguage queryKind, String source, IEnumerable`1 imports, IEnumerable`1 references, IndentingLineWriter writer) in A:\LinqPadless\src\Program.cs:line 558
   at LinqPadless.Program.<>c__DisplayClass5_0.<Compiler>b__0(String queryFilePath) in A:\LinqPadless\src\Program.cs:line 280
   at LinqPadless.Program.Wain(IEnumerable`1 args) in A:\LinqPadless\src\Program.cs:line 185
   at LinqPadless.Program.Main(String[] args) in A:\LinqPadless\src\Program.Main.cs:line 33

The compilation was indeed unsuccessful because the C# compiler emitted the following error:

LinqPadless-Issue-13.cs(23,15): error CS0234: The type or namespace name 'CodeAnalysis' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?)

I suspect that the following warning has something to do with (i.e. no references found):

  Warning! Packages with no references for .NETFramework,Version=v4.5.2:
    Microsoft.CodeAnalysis.CSharp.Workspaces 2.0.0
    Microsoft.CodeAnalysis.CSharp 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0
    Microsoft.CodeAnalysis.VisualBasic.Workspaces 2.0.0
    Microsoft.CodeAnalysis.VisualBasic 2.0.0
    Microsoft.CodeAnalysis.Common 2.0.0
    Microsoft.CodeAnalysis.Workspaces.Common 2.0.0

Since most of the libraries in those packages target .NET Standard 1.3, I also tried invoking lpless with the --fx=netstandard1.3 option but the end-result was the same.

@vcaraulean
Copy link
Contributor

Indeed, without specifying the target, the default compilation target of linqpadless is used, so it's net452. As @atifaziz mentioned, libraries are targeting netstandard1.3 which is not covering 4.5.2. Closest target is 4.6.

Patch from #17 makes correct use of target framework ('-fx' key). Results of testing:

-fx=net462 or -fx=netstandard1.3

  • compiles fine
  • .cmd gives runtime error about missing types

Error details:

System.AggregateException: One or more errors occurred. ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Composition.TypedParts, Version=1.0.27.0, Cultu
re=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.                                                                  
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)                                                                                               
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.get_DefaultHost()                                                                                                              
   at Microsoft.CodeAnalysis.AdhocWorkspace..ctor()                                                                                                                                  
   at Submission#0.<<Initialize>>d__0.MoveNext()                                                                                                                                     
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()                                                                                   
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.Script`1.<RunSubmissionsAsync>d__21.MoveNext()                                                                                                
   --- End of inner exception stack trace ---                                                                                                                                        
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)                                                                                                
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunScript(ScriptOptions options, String code, ErrorLogger errorLogger, CancellationToken cancellationToken)         
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunInteractiveCore(ErrorLogger errorLogger)                                                                         
   at Microsoft.CodeAnalysis.Scripting.Hosting.CommandLineRunner.RunInteractive()                                                                                                    
   at Microsoft.CodeAnalysis.CSharp.Scripting.Hosting.Csi.Main(String[] args)                                                                                                        
---> (Inner Exception #0) System.IO.FileNotFoundException: Could not load file or assembly 'System.Composition.TypedParts, Version=1.0.27.0, Culture=neutral, PublicKeyToken=b03f5f7f
11d50a3a' or one of its dependencies. The system cannot find the file specified.                                                                                                     
File name: 'System.Composition.TypedParts, Version=1.0.27.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'                                                                       
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)                                                                                               
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.get_DefaultHost()                                                                                                              
   at Microsoft.CodeAnalysis.AdhocWorkspace..ctor()                                                                                                                                  
   at Submission#0.<<Initialize>>d__0.MoveNext()                                                                                                                                     
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()                                                                                   
--- End of stack trace from previous location where exception was thrown ---                                                                                                         
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)                                                                                                      
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                 
   at Microsoft.CodeAnalysis.Scripting.Script`1.<RunSubmissionsAsync>d__21.MoveNext()                                                                                                
                                                                                                                                                                                     
WRN: Assembly binding logging is turned OFF.                                                                                                                                         
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.                                                             
Note: There is some performance penalty associated with assembly bind failure logging.                                                                                               
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].                                                                                      

@vcaraulean
Copy link
Contributor

The type in case is residing in Microsoft.Composition package. But there's #20

@atifaziz
Copy link
Owner

atifaziz commented Mar 28, 2017

With #16 and #20 resolved, the compilation now fails due to the following error from the C# compiler:

LinqPadless-Issue-13.cs(22,9): error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=4.0.20.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

That happens when targeting .NET Framework 4.6. When targeting 4.6.2 instead, compilation passes but running the executable throws the following error:

Unhandled Exception: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeAssembly.get_DefinedTypes()
   at System.Composition.Hosting.ContainerConfiguration.<WithAssemblies>b__0(Assembly a)
   at System.Linq.Enumerable.<SelectManyIterator>d__16`2.MoveNext()
   at System.Composition.TypedParts.TypedPartExportDescriptorProvider..ctor(IEnumerable`1 types, AttributedModelProvider attributeContext)
   at System.Composition.Hosting.ContainerConfiguration.CreateContainer()
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable`1 assemblies)
   at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.get_DefaultHost()
   at Microsoft.CodeAnalysis.AdhocWorkspace..ctor()
   at UserQuery.Main()

The LoaderExceptions property had one FileLoadException entry whose Message property read:

Could not load file or assembly 'System.Collections.Immutable, Version=1.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

and FusionLog property read:

=== Pre-bind state information ===
LOG: DisplayName = System.Collections.Immutable, Version=1.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
 (Fully-specified)
LOG: Appbase = file:///B:/
LOG: Initial PrivatePath = NULL
Calling assembly : System.Reflection.Metadata, Version=1.4.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: B:\lpless-issue-13.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: The same bind was seen before, and was failed with hr = 0x80131040.

The System.Collections.Immutable package version installed during compilation was 1.3.1. If some assembly in the dependency chain has a reference to version 1.2.0 then the exception is understandable (and because lpless doesn't add any assembly binding redirects).

The actual version of System.Collections.Immutable.dll in the 1.3.1 package is 1.2.1. Adding the following redirection to the compiled executable's configuration file made the executable finally work as in LINQPad:

<dependentAssembly>
  <assemblyIdentity name="System.Collections.Immutable"
                    publicKeyToken="b03f5f7f11d50a3a"
                    culture="neutral" />
  <bindingRedirect  oldVersion="0.0.0.0-1.2.1.0"
                    newVersion="1.2.1.0" />
</dependentAssembly>

It seems then that lpless needs to add automatic binding redirections as NuGet would. How would this ever work for a C# script? 🤔

@vcaraulean
Copy link
Contributor

I remember dealing with similar issues. If there's no API level compatibility issues (missing types, signatures, members) dynamic assembly resolution might work. A good example can be found in somewhat related issue for scriptsc. And implementation. Thoughts?

@atifaziz
Copy link
Owner

@vcaraulean Thanks for the scriptcs pointers, and yeah, I thought of dynamic assembly resolution too but at this stage things seem to be getting quite complicated and I'm wondering if the scenario of compiling to C# script versus an executable is buying anything extra. After all, it's not like the C# interpreter is readily downloadable from anywhere (even keeps moving locations and redistribution channels) so it's only good for environments where the build tools are installed.

With NuGet integration, compilation and more, I fear that LINQPadless might end up replicating the complexity of the build process and artifacts (developer packs, references per target framework, forwarded types, facades) & package installation. @wangzq took an interesting approach with lpmake where he compiles the LINQPad query to a project and then leverages the build tools to do all the heavy-lifting. With LINQPadless, the goal was having few files to manage and distribute (*.cmd + *.csx or *.cmd + *.exe with auto-restore at target if packages are missing) and least tooling (e.g. NuGet and MSBuild are not required). Just thinking out loud here for now. 💭

@wangzq
Copy link
Author

wangzq commented Mar 29, 2017

This issue is really about how to get correct assembly references from a nuget package for a specific framework version. This used to be easier with just .NET 4.x, with NetStandard it is getting more complicated. In lpmake I am relying on project.json to figure out the correct references for me, but it also stopped working for recent packages such as 2.0 of Microsoft.CodeAnalysis. My plan is to either learn more about how Linqpad is able to do that, and/or check if the new msbuild is able to solve this problem.

@atifaziz
Copy link
Owner

After all, it's not like the C# interpreter is readily downloadable from anywhere (even keeps moving locations and redistribution channels) so it's only good for environments where the build tools are installed.

Looks like C# interpreter is available as part of Microsoft.Net.Compilers so doesn't require a Microsoft Build Tools 2015 installation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants