Provide MSBuild resolution instead of direct file references for transitive dependencies

Topics: General
Editor
Mar 14, 2013 at 11:59 AM
After the discussion here: https://nuget.codeplex.com/discussions/433942

I thought it might be worth starting a direct discussion about not adding transitive references directly to a .csproj. Although the current approach does work, it is pretty untidy. Instead, I have put together a proof of concept (bit rough around the edges) that shows only direct references being added to the .csproj, with the tranistives being found and copied via an addition to the AssemblySearchPaths property in MSBuild.

Proof of concept found here: https://github.com/BenPhegan/NuGetAutoTransitiveDependencies

Is it worth looking at providing this as an option within NuGet? This would require (at least):

1) A .csproj to have a new msbuild <import> statement injected.
2) An option to not add transitive references as file references when adding a package.

Thoughts?
Developer
Mar 14, 2013 at 4:54 PM
Sorry for my ignorance, but what is the problem you are trying to solve here? And what is a transitive reference?
Mar 14, 2013 at 6:03 PM
Hi dotnetjunky,

At least one aspect of the problem at hand is the management of assembly references when the package dependency tree is deep.
In my organisation, it's quite common that a project will depend on only 2 or 3 packages, but those packages bring in a dozen of additional dependencies.

Those additional, indirect dependencies are the one which introduce what BenPhegan called 'transitive references', as all their assemblies are automatically referenced in the csproj by nuget.

I'd like to avoid that because it clutters the project: it becomes impossible to see which assemblies are really used in the project code.

But even though the assembly references aren't needed at build time, they currently cannot be removed because they might be required at run time.
Hence we're looking for a way to make it work at runtime without explicit assembly references in the csproj.

Please let me know if this is clear enough, I might not be using the right words for some concepts.
Editor
Mar 14, 2013 at 9:05 PM
Thanks for clarifying, @broggeri. Within our organisation, adding a single package can result in over 100 additional packages (and yes,this is a separate problem!) being added to the .csproj as references, although they are really only required for runtime. This fix addresses the requirement for direct file references in order to get a functional runtime output.
Mar 14, 2013 at 11:26 PM
I'm curious if the "bin deployable" feature talked about in various other articles is related enough to dig through. For example, Phil Haack mentions bin deploying for .NET MVC via a special "_bin_deployableAssemblies" folder in a project. The same thing is outlined in one of Scott Hanselman's articles.

Now, it seems that using the special folder would still require a bit of management by NuGet/MSBuild. The technique described by BenPhegan seems to avoid that hassle of it entirely. However, would this technique work with Web Deploy? Currently, only referenced assemblies or assemblies found in the special _bin_deployableAssemblies folder are deployed, AFIK.
Mar 15, 2013 at 10:06 AM
Hi @BenPhegan,
Looking at https://github.com/BenPhegan/NuGetAutoTransitiveDependencies/blob/master/AutomaticTransitives.targets, I see that all assemblies in the repository are made available in the assembly search path.

However, a repository may be shared between several projects in the same solution, and may contain several versions of the same packages.

In order for the build system to choose the right version, it would have to select the right version based on the project's packages.config file.

Is there enough flexibility in the msbuild language to replace:
<NuGetSearchPathFiles Include="$(SolutionDir)packages\**\*.dll" />
by something which would list explicitly the package directories based on the packages.config file ?
Editor
Mar 17, 2013 at 10:58 AM
@broggeri for transitive assemblies, having multiple of the same assembly under packages is not a problem. I have updated the example to show that:

1) HintPaths are not required.
2) With four versions of Castle.Core in the packages directory, the correct one is chosen (this shouldn't be a real surprise, otherwise we wouldn't have binding redirects)

One last thing that I haven't shown is the effect of different assembly names on the resolution of the correct assembly. Will hopefully push that up this week if I get a chance. This is a little more interesting, as I believe that if you have a reference as simple as "Castle.Activerecord" then it might be a little hard to tell which you will get. If you use a fully qualified assembly name such as "Castle.Activerecord, Version=2.5.1.0, Culture=neutral, PublicKeyToken=null" then you should get the correct one, regardless of how many are available under packages.

There is a caveat to this technique. If you have a really large number of paths under packages, then each will be checked for each assembly being evaluated without a hint path. This will have a performance hit on the build. It should not be hugely significant, but it will be there.
Mar 18, 2013 at 6:09 AM
Mar 18, 2013 at 2:39 PM
@BenPhegan:

Somehow I believe that it would still be worth it to restrict the assembly search path to only the folders containing the correct versions, because:
  • Using the fully qualified assembly name means that the information from the packages.config file is duplicated, which is a bit unfortunate.
  • While in simple cases, the correct version is chosen, I'm afraid of non-deterministic results in more complex cases. Note that binding redirects aren't always necessary for a different version of a dll to be picked up (I think they might be only required for strongly named assemblies, I'm not exactly sure though).
example :
  • A depends on C >= 1.0.0, B depends on C >= 1.0.0,
  • the project references A and B directly, and packages.config contains C 1.0.2.
  • A was built with C 1.0.0,
  • B was built with C 1.0.1.
  • for some reason, the repository contains C 1.0.0, 1.0.1 and 1.0.2
The chosen the version will be indeterministic (it will depend on the order of link resolution), and it will be either 1.0.0 or 1.0.1, while the expected result would be 1.0.2.

If we could restrict the search path to only the 1.0.2 folder, we would get the expected result in a deterministic way, and if for some reason, C 1.0.2 was missing (maybe because the repository has been corrupted or any unexpected reason), the link would fail explicitily (instead of silently fallbacking to another available version, which I'd rather avoid - again for avoiding indeterministic results down the road).

I guess if I'm so convinced of those issues I should come up with a nice test case / POC as you did :)
But I've yet to find out how hard would it be to use the information from the packages.config in a msbuild script...
Mar 21, 2013 at 3:57 PM
I tested a small improvement over the POC in https://github.com/BenPhegan/NuGetAutoTransitiveDependencies/blob/master/AutomaticTransitives.targets (unrelated to my previous comment).

When mixing this with the package restore targets, it doesn't work well because the NugetAssemblyFile pattern is evaluated before the package are restored.

I tried nesting the ItemGroup inside of the BeforeResolveReferences Target: it works.
Editor
Mar 26, 2013 at 10:24 AM
Cool, did you fork/commit? If so, happy to send a pull request? Or should I just modify and push it up if that was the only change?

There are probably a bunch of other edge cases that could be clarified around this, however all cases of simple transitive checking that I have tried seem to resolve correctly. I havent had a lot of time on this recently though, so will try and build out some of the cases over the weekend. Also interested to see what the NuGet core team are thinking, as this will require either a fork or a nasty hack to NuGet for it to be integrated well...
Mar 26, 2013 at 11:17 AM
@BenPhegan we've been playing around with this in our ProGet Client Tools as well. Here's an example of a packages.config:

<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="InedoLib" version="5.1.*" copyTransitiveDependencies="true" />
</packages>

The copyTransitiveDependencies flag instructs the tools to do basically what you're doing: copy the transitives' DLLs to the {projectOut} directory at SYNC (from within Visual Studio) or INSTALL (the .exe). We don't like adding pre-build things to the project automagically, so it'd be nice to to make it auto-SYNC on build, or something.

Obviousy this renders packages.config incompatible with NuGet's Client tools, but that lets us do the 5.1.* thing. As an enterprise NuGet user, would this approach be helpeful?