PROTOTYPE Implementation of Native Libraries in NuGet

Topics: General
Dec 20, 2012 at 9:35 PM

THIS IS A PROTOTYPE. DON’T USE THIS FOR ANY ACTUAL WORK.

I’ve reached what I consider the end (at least for now) for my prototype of adding support for native packages (via the Extension SDK mechanism http://msdn.microsoft.com/en-us/library/vstudio/hh768146.aspx) into NuGet. The original specification was defined at http://sdrv.ms/WxwS40.

I’m temporarily finished for a number of reasons. First is that I’m reaching the end of the time my employer has allocated for me to work on this. Second I think this is at a point where decisions can be made by the community as to whether this is the proper way to move forward with native libraries in NuGet.

How to Use:

  1. Get my fork of NuGet from http://nuget.codeplex.com/SourceControl/network/forks/wwahammy/NuGetWithNative and press Run in VS. This should open a new VS2012 window with the custom version of NuGet

    NOTE: You must have uninstalled all other versions of NuGet including the one that comes with VS.
  2. Download the sample project from http://sdrv.ms/WxwIcH, unzip and open it in your new VS2012 window.
  3. Download the SampleLib NuGet Package from http://sdrv.ms/WxwO4b. Make sure this package is in a local package repository.
  4. Add the SampleLib package to the Win32Project1 project like normal. Once installed, you should now be able to build and run Win32Project1. (The needed headers will be referenced, the library paths will be modified, the lib file will be added to the linker and the Dll will be copied to the output directory of Win32Project1 at build time)

Known bugs:

  • I’ve ignored the Extension SDK+ part of the specification. This is a prototype and it’s really a substantial amount of work.
  • Uninstall misses some stuff added to the project file such as taking out the Imports for the props files and removing the SDKReferenceDirectoryRoot when it is no longer necessary.
  • There’s no way to know if a package is applicable to a given project based upon platform, compiler type, etc.
  • When a props file import is added to your project, it’s always referring to the package repository as if it’s in the solution. If it’s in a different location, you’re out of luck.
  • The props file that comes with my SampleLib.1.0.0.nupkg file will always copy over the Debug dll to the project output folder, never the Release.
  • The extension SDKs don’t show up in a references node, like they’re supposed to. I have no idea why. I think it has to do with the fact the project is reporting back a platform version of 7.0 when Visual Studio expects one of no lower than 8.0.

My opinion:

I am of the opinion that the Extension SDK mechanism just isn’t right for native packages in NuGet. It simply has too many limitations some of which follow:

  • Dependency mechanism that replicates one in NuGet
  • Expects every Extension SDK to be a child of a platform SDK. If your library isn’t really dependent on any particular platform, you’re out of luck.
  • Native libraries come in different flavors that Extension SDK was never meant to handle. Builds for different versions of Visual C++ or another compiler, static vs. dynamic libraries, etc.
  • You need an SDKManifest.xml file which replicates much of the functionality of a .nuspec file.
  • You have to have a props file to do all the various imports for headers, import libraries or dlls. This could be done via one of the powershell files as part of the NuGet installation process or better yet, built into the NuGet itself so when you install a Native library, it handles all these common import functions just like you would for .NET assemblies.
  • Extension SDKs handle things like adding tools to the Visual Studio UI. This isn’t needed by nearly any native packages. If I just want a version of zlib, I don’t need that.

Extension SDKs seem to have been made for one reason: Windows Store. The problem is NuGet works already for Windows Store (as long as it’s a .Net project) and does it in a more straightforward way. Extension SDKs don’t seem like a generalized solution to library packaging problems like NuGet. Instead of going down this path further with Extension SDKS, I recommend with minor modifications to NuGet we can support native libraries in all cases without trying to smash together two packaging mechanisms that weren’t intended to ever work together.

Next steps:

While I've stated my opinion, I don’t decide this; the community does. I recommend people evaluate the prototype, play with the code, think through these issues and discuss this all on here over the next few weeks. I have some ideas in mind in how we go forward on this if we choose not to go with using Extension SDKs but I’d like everyone else’s ideas too. Please respond to this thread with thoughts, concerns, ideas, just anything remotely relevant. No matter where we move forward on native libraries, I’m confident we’re very close to a solution in NuGet.

Dec 21, 2012 at 8:25 PM

  • Expects every Extension SDK to be a child of a platform SDK. If your library isn’t really dependent on any particular platform, you’re out of luck.
  • Native libraries come in different flavors that Extension SDK was never meant to handle. Builds for different versions of Visual C++ or another compiler, static vs. dynamic libraries, etc.

These two issues are for me a complete show stopper for this approach. From what I gathered from the Extension SDK docs, this is a major shortcoming which shows how it wasn't meant for this purpose at all.

The c++ support I had implemented addressed all the issues your prototype still has, and definitely took into account the different linkage scenarios of the CRT.

What was the main motivation behind using the Extension SDK mechanism :

  1. reuse build in visual studio facilities to support native libraries ?
  2. or make vsix packages compatible with nuget ?

If the answer is 2. then it could make sense to provide a vsix to a nuspec conversion tool. Most of the information is already available in the vsix, it must just be extracted and probably extended a bit before producing a nuget package.

I'll have a go at your prototype to get a feel of how practical or not this is for the libraries I have in mind.

Dec 21, 2012 at 8:40 PM

TeaJay: I can't speak for the people involved in the original planning, so they'd need to speak for themselves. To me it it seemed to be nice to reuse functionality already in Visual Studio and it'd be nice to make Extension SDKs available in NuGet. Extension SDKs also have the ability to modify the UI in Visual Studio. For example, it could add toolbox items for functionality that an installed Extension SDK provides.

If you have any documentation on you implemented libraries, could you post that? I'd like to get some other viewpoints on how best to solve this problem.

Dec 22, 2012 at 1:31 AM

Nuget could already fully support the Extension SDK. Here's how:

  1. Pack the extension SDK in a nuget package
  2. Add an install/uninstall script to add/remove the necessary <SDKReferenceDirectoryRoot> entry.

I think supporting the extension SDKs or making Nuget compatible with the extension SDKs shouldn't get in the way of adding full support for native C++ projects.

If it really is the intention to bridge the gap between Nuget and the Extension SDKs fine. But for me it's a completely different matter. Supporting the extension SDK would also impact the whole functionality of Nuget as the Extension SDKs purposes aren't merely to provide native C++ support. Extension SDKs also support all the .NET languages.

So, I'm really not convinced yet it's the right way to go.

Developer
Dec 31, 2012 at 9:45 PM

First of all, thanks wwahammy for spending your time working on this prototype. Great effort. I've tested your fork and it looks good. I have not looked closely at your code though. Do you want me to comment on the code or do you think it's better that we discuss on the issues that you listed first?

Now I can speak for the NuGet team as to why we chose to adopt the extension SDK as a vehicle to support native libraries.

When we had a meeting with the team who owns the Extensions SDKs, it is our understanding that Extensions SDKs provide full support for native projects. That's why we decided to explore this path, because if implemented successfully, it would save us a great amount of effort to support native projects directly. I know you, teajay, has worked on this in your fork for a while (and probably has got it working well), but we have to account for testing and maintenance effort in the long run too.

That said, I want to address the limitations that wwahammy pointed out:

  • Dependency mechanism that replicates one in NuGet

    Can you clarify? AFAIK, Extensions SDKs do not support dependency mechanism.

  • Expects every Extension SDK to be a child of a platform SDK. If your library isn’t really dependent on any particular platform, you’re out of luck.

    If my library isn't dependent on a particular platform, will it work if I state that it depends on Windows?

    Native libraries come in different flavors that Extension SDK was never meant to handle. Builds for different versions of Visual C++ or another compiler, static vs. dynamic libraries, etc.

    So what is really supported by Extensions SDK? If you can list the things that Extension SDKs miss, I'll ask the Extensions SDK team to seek their opinion.

  • You need an SDKManifest.xml file which replicates much of the functionality of a .nuspec file.

    I'm not so sure about this. The nuspec's main functionality is to describe the metadata of the entire package. The SDKManifest.xml file, on the hand, only describes the file list to instruct VS how to use the SDK. I don't see much replication here.

  • You have to have a props file to do all the various imports for headers, import libraries or dlls. This could be done via one of the powershell files as part of the NuGet installation process or better yet, built into the NuGet itself so when you install a Native library, it handles all these common import functions just like you would for .NET assemblies.

    Why is it a limitation? It's part of the Extensions SDK model. We don't expect there will be many native library packages, so this will not be a big issue.

  • Extension SDKs handle things like adding tools to the Visual Studio UI. This isn’t needed by nearly any native packages. If I just want a version of zlib, I don’t need that.

    AFAIK, the tooling aspect of Extensions SDK is optional. If you don't need it, you don't have to add it to the SDK. Or am I missing something?

  • Developer
    Dec 31, 2012 at 9:49 PM

    I forgot to add that besides adding support for native libraries, our goal with Extensions SDKs is to also implement proper support for Windows Store app flavors, as being discusses in this topic:

    http://nuget.codeplex.com/discussions/402786

    Jan 2, 2013 at 7:07 PM
    Edited Jan 2, 2013 at 9:40 PM

    We probably should just discuss the issues. The code isn't all that nice but there's probably things there you could use in the future.

    • Can you clarify? AFAIK, Extensions SDKs do not support dependency mechanism.

    The first dependency mechanism is on whether the platform SDK is available. Extension SDKs are, as the name implies, an extension of a platform SDK. This is described through the folder structure. This is a dependency for development only. A project were to depend on the Extension SDK, they wouldn't need anything from the relevant platform SDK for deployment. It's a little like a Framework Assembly in NuGet now (but of course, it's inconsistent).

    The second dependency mechanism is the SDKs described in the DependsOn attribute on the root FileList element of an SDKManifest file. I think these are supposed to be Extension SDK identifiers but the documentation isn't really clear. (http://msdn.microsoft.com/en-us/library/vstudio/hh768146.aspx)

    In short, Extension SDK has two dependency description mechanisms, which are described in separate places and aren't consistent with NuGet.

    • If my library isn't dependent on a particular platform, will it work if I state that it depends on Windows?

    No. The Extension SDK directory structure requires a platform identifier (Windows) and a platform version (v8.0). If you know it will work on any version of Windows, you'll have to create a platform version folder for every version with duplicate contents in each folder. This sorta happens now with NuGet since you can't easily say I want to support a version range (I think) for a framework but you do have the option of saying I support all versions of a given framework.

    • I'm not so sure about this. The nuspec's main functionality is to describe the metadata of the entire package. The SDKManifest.xml file, on the hand, only describes the file list to instruct VS how to use the SDK. I don't see much replication here.

    It's a second file that describes how a package is used. That's replication. It won't make sense to people making packages. It also describes things related to the name such as DisplayName. Duplicated effort for little gain.

    • Why is it a limitation? It's part of the Extensions SDK model. We don't expect there will be many native library packages, so this will not be a big issue.

    There won't be many native library packages if it's too cumbersome to make them. If NuGet does native packages in a way that is as easy for managed packages, people will love it and use it for native development on Windows.

    To understand why it's a limitation, imagine having to not just put in the assembly for your package but describe its interface multiple places and then figure out how to refer to the props file since it could be in multiple places depending on where NuGet places the package. This is silly because in almost all cases native libraries require two things: headers and import libraries. (if the library is dynamically linked, you need the DLL for deployment and testing)

    NuGet is pretty big on convention over configuration. That's why you don't have to specify which .net frameworks you support in your nuspec file, or that if you want to get the binaries for the package you go into the "lib" folder. NuGet just does what you'd expect it to do.

    There’s no reason NuGet can’t do the same for native packages using the following directory structure.

    lib
       vc80+dll (this library built for vc 8.0 and is dynamically linked)
          headers
             <headers for the library>
          lib
            <import libraries for the library>
         <DLL, symbols, etc for the library>

    This structure represents most open source libraries available. When installing a package, NuGet can add the headers directory path to the include path in the VC++ project, add inner lib directory path to the import directory paths in the VC++ project and add each of the .lib files to the linker import libraries setting in the VC++ project. For more complex packages, with .tlb files and special compilation settings, a main.props file could be put in the vc80+dll folder. NuGet would add an import statement into the VC++ project file for this main.props file if it’s there. Even more complex package installation functionality (Extension SDK) could be scripted using the init.ps1 file.

    NuGet’s success is in convention over configuration. The intent of convention over configuration is to make the common situation easy and the less-common situation possible. Extension SDK is not a general packaging mechanism like NuGet, it’s a specialized solution to a particular problem. Making all native library packages go through the effort of creating an Extension SDK makes the common situation hard and the less common situation hard as well. It’s shoehorning a truly a specialized solution (with no uptake outside of MS) into a general solution slot.

    I think we have a vital opportunity to make native development library usage on Windows as easy as managed libraries. I don’t believe Extension SDK is the way to go about doing this though.

    Developer
    Jan 3, 2013 at 6:20 PM

    The first dependency mechanism is on whether the platform SDK is available. Extension SDKs are, as the name implies, an extension of a platform SDK. This is described through the folder structure. This is a dependency for development only. A project were to depend on the Extension SDK, they wouldn't need anything from the relevant platform SDK for deployment. It's a little like a Framework Assembly in NuGet now (but of course, it's inconsistent).

    The second dependency mechanism is the SDKs described in the DependsOn attribute on the root FileList element of an SDKManifest file. I think these are supposed to be Extension SDK identifiers but the documentation isn't really clear. (http://msdn.microsoft.com/en-us/library/vstudio/hh768146.aspx)

    In short, Extension SDK has two dependency description mechanisms, which are described in separate places and aren't consistent with NuGet.

    I see. I didn't know that there's a DependsOn attribute. I understand your concern about this, but honestly though, I don't see it as a big problem. For most native libraries, there won't be dependencies on another SDKs, so this attribute won't be needed.

    No. The Extension SDK directory structure requires a platform identifier (Windows) and a platform version (v8.0).

    This is indeed a deal breaker. Let me contact the Extensions SDK team to ask them to comment on this.

    For the rest of your comment, I generally see your point. I'll discuss with the rest of the team and will get back to you.

    Overall, thanks for your effort so far.

    Jan 6, 2013 at 8:32 PM

    Thanks wwahammy for putting the concerns into words. My wording was a bit short :)

    Developer
    Jan 7, 2013 at 8:58 PM

    Hi,

    I've talked to the Extensions SDK team and verify with them that the current Extensions SDK feature only supports Windows Store project, and hence it can only work on Windows 8.

    So, Eric, we agree with your comment that Extensions SDK is not the best way to support native libraries.

    This means in order to support native libraries, we will have to do it directly. teajay, I believe you have been working on this for a while, correct? Can you share with us the current state of your fork? Is it fully working or it still significant amount of work?

    Jan 8, 2013 at 7:42 AM
    Edited Jan 8, 2013 at 7:46 AM

    I haven't followed the latest development, my fork is still based on Nuget 1.8. I'll have to bring it up to the current state which shouldn't take too long. 

    There is still one improvement I need to implement which is to add a check for include paths pointing outside of the nuget package directory structure when building a package. This is just a package developer helper function which shouldn't block any early testing of the new functionality. I've tested the extension on some of my projects but broader feedback would be most helpful.

    I'll let you know as soon I get my fork updated.

    Developer
    Jan 8, 2013 at 10:02 AM

    Thanks teajay. Let me know if you need help with rebasing your fork/branch on top of the latest.

    Jan 9, 2013 at 2:45 PM

    I've come pretty far already with the update. There are still a couple of rough edges I need to smooth out ... 

    But I'v hit a big issue I don't know how to solve:

    For the Visual C++ project support I need to access the VCProjectEngine and VCProject classes of the Visual Studio SDK. The problem here is that the assembly version absolutely needs to match the visual studio version for the binding to work and the assembly names are identical !

    The assembly names are Microsoft.VisualStudio.VCProjectEngine.dll and Microsoft.VisualStudio.VCProject.dll.

    To support both visual studio versions (Vs2010 and Vs2012) i would need a mechanism to switch the referenced assemblies during runtime and not build time. The assemblies are referenced in the VisualStudio and in the Core projects. So this poses a problem for the Command line tool and for the visualstudio extension.

    If anyone knows a trick ... 

    Developer
    Jan 9, 2013 at 6:39 PM

    Hi teajay,

    Please hold off on this. wwahammy and I had a conversation this morning about this. He and another MS employee cam up with a much simpler solution to supporting native libraries. In a nutshell, we will require the package authors to put every required files under a \native folder inside the .nupkg file, together with a .props file. When NuGet installs the package, it will add the .props file to the project file and the .props file will take care of everything.

    I'm sorry for this last minute change. I'd appreciate your effort. Please feel free to add your comments & thoughts on this approach.

     

    Jan 9, 2013 at 8:33 PM

    I though about this solution also except writing the props file isn't going to be much fun ... and ... it doesn't solve half of the user experience issues which is what I'm actually working on :

    • how do you give the user feedback that the package is compatible with the project type ? You have to check that the CRT version matches between the package and the application if the interface is not a simple C interface. Putting everything in a native directory and hope that there will be something compatible with the c++ project is not very helpful.
    • how do you add .net libraries to managed c++ project ? We should also support adding .NET assemblies to managed C++ projects without needing to rewrite existing nuget packages.
    • for a package author, writing a props file isn't going to be any fun and not a trivial task. You have to dive deep into the MSBuild core constructs, understand that correctly and implement that correctly in order for everything to work. The approach I've taken is on this point much simpler and conform to the actual nuget package authoring experience. 

    The idea isn't bad, except it doesn't solve half of the problem. What it can simplify however is the way new includes and defines or compiler options can be added to the project. That would simplify the nuget code a bit and make reverting the changes made by nuget much more robust.

    Coordinator
    Jan 9, 2013 at 9:56 PM

    @teajay What do you think about our starting out with the props file convention to unblock the scenario for the libraries that could provide the props file, but then considering adding more functionality directly into NuGet in a later release?

    Developer
    Jan 9, 2013 at 10:05 PM

    @teajay: To answer your question #3, we'll provide a command tool to allow package authors to produce the props file easily.

    Jan 10, 2013 at 6:19 AM
    Edited Jan 10, 2013 at 6:28 AM

    @jeffhandley: If in a first step adding a props file is sufficient I would only enable the powershell scripts for c++ projects and wouldn't change anything else for now. With the powershell script adding a prop file is very easy. Providing a sample script would get any developer wanting to provide a package in this form going. That leaves room for any further development without breaking any packages.  It also give the necessary flexibility to author complex packages.

    @dotnetjunky: That means you would derive the props file from the nuspec file or from some other source ? I was thinking the props file could be generated on the file if none is provided. This file could also be written out to disk with a special command to serve as a starting point for package authors.


    Any how I got things working again based on nuget 2.3, only for VS2012 due to the binding issue I mentioned above.
    You can check it out here: https://git01.codeplex.com/forks/teajay/868cppprojectsupport2 


    The package compatibility checking works pretty well, at least with all the packages I tested. Replacing the direct extension in the project file of the compiler options by the usage of a props file shouldn't be that complicated.
    If you need any sample packages there are some in the issue 868. 

    Please let me know what you think of this, and tell my how I can continue to help to push this on.

    Jan 10, 2013 at 5:07 PM

    @teajay ->

    Next week, I'll be writing a tool/cmdlet that will generate the .props file. (and generate you a .nuspec too, if you didn't have one).

    Shouldn't take me too terribly long, I've got code lying around that does most of it already.

    Jan 11, 2013 at 4:07 PM

    I'd be glad to check this out !

    Jan 17, 2013 at 2:49 AM

    There's another aspect of this that I just realized/discovered.

    Windows Phone 8 apps that use Native components are also affected, so it's not just Windows Store apps.

    Here's the issue: If you use a native lib in your phone app, then you need to provide both x86 and ARM configurations. x86 for the emulator and ARM for the device. This isn't an issue for "pure" .NET assemblies as they can be AnyCPU.

    WP8 appears to use the same Extension SDK mechanism to handle having binaries for both x86 and ARM in the same package so that VS can use the right one based on the project/solution configuration.