NuGet & some dependency version questions

Nov 3, 2010 at 8:57 PM

Full disclosure:  I'm coming from the Java side of the world, with much experience in Maven (for project source layout, transitive dependency management, unit & integration testing, and builds) and Ivy (transitive dependency management only), as well as some experience in OSGi (class loader isolation).

Some questions:

  1. How do you specify a range of versions of a dependency?  For example, how do I say that I depend on any version 3 of package foo?  In Maven, you use parentheses (exclusive) and square brackets (inclusive) to specify the version range (see Maven version range specifications):
    <dependency id="foo" version="[3,4)" />

    The above would mean that I depend upon any version of package "foo" from and including 3.0.0.0 up to but not including 4.0.0.0.  This would mean, by the way, that

    <dependency id="bar" version="3.0.0.0" />

    is shorthand for

    <dependency id="bar" version="[3.0.0.0]" />
  2. Are versions "3", "3.0", "3.0.0", and "3.0.0.0" all considered equivalent versions?
  3. Does version 3.12 (minor update number twelve of major version three) compare greater than version 3.2 (minor update two of version three) as it should?
  4. (This one's a doozy)  What do you to when two dependencies that you add to a project directly or transitively refer to incompatible versions of the same dependency?  For example, let's say my package, foo, directly depends upon package goo version 1.2 and package koo version 2.2.  Additionally, it turns out that package koo depends (directly or transitively, it doesn't really matter) on goo version 2.0.  To really gum up the works, goo version 2.0 is not backward-compatible with goo version 1.2 -- there are breaking changes, and my stuff really needs goo 1.2 and koo really needs goo 2.0.  So I'm stuck.  FYI, this is exactly the problem that OSGi solves today in Java land.

Thanks,

Matthew

Coordinator
Nov 3, 2010 at 9:58 PM

1. Right now we you either supply an exact match version attribute, or you use a combination of one or both of minVersion and maxVersion attributes. However, I do like the maven approach you linked to: http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html

We’re also considering supporting some aspects of a Semver based approach: http://nuget.codeplex.com/discussions?searchText=SemVer

For example, we may consider version="1.2" to be equivalent to "1.2.*". Whereas version="1.2.1" is equivalent to 1.2.1.* and 1.2.1.3 is an exact match. So we treat the number of octects in the version you give us as a measure of the precision you care about.

2. Not under our new proposal, no.

3. It should. Need to test that.

4. We have a bug against that, though we’re not sure if we’ll get it into v1. Currently we fail with a conflict. We plan to allow for conflict resolution and perhaps automatic dependency version leveling in cases that are “safe”. http://nuget.codeplex.com/workitem/296

How does OSGi solve today? BTW, it’s great to have someone with Maven experience involved so please do look through our issues and discussions and chime in. J I’ve tried to highlight the relevant discussions as best as I can.

Developer
Nov 4, 2010 at 2:26 AM

We're very passive right now in the way we deal with dependencies. Any feedback in this area would be great!

Nov 4, 2010 at 1:28 PM

1. I like the maven approach, too, but then, I'm biased.  :)  The problem with the minVersion & maxVersion approach is that if you want to exclude a particular within the range that was bad, you can't.  For example, if foo 1.0.0 was good, 1.0.1 had a bug, and 1.0.2 fixed it, and you're confident that the foo committers won't screw up like that again while still working on the first major version, you could specify the version range as two sets of ranges, as shown in the maven docs.

2. I would recommend that 3 == 3.0 == 3.0.0 == 3.0.0.0, because it would otherwise bring ambiguity to how versions are compared.  Making 3 < 3.0 < 3.0.0 < 3.0.0.0 be true seems counterintuitive to me, and version comparisons need to be pretty straightforward.

3. That's a relief.

4. I think that this is not a solvable problem in .NET (or Java, for that matter, which is partly why OSGi came into being).  I think .NET might not let you get past this problem at build time, thanks to the first-class treatment of assemblies.  In Java, there is no direct correlation to a .NET assembly; the closest that there is in plain old Java is a JAR, but it doesn't have the version information that a .NET assembly must have (by the way, I love the "internal" keyword, and wish Java+OSGi had it).  In OSGi (that is, Java running in an OSGi environment), there is a notion of a "bundle", which adds three pieces of information to a jar:  its unique id (comprised of groupId & artifactId), the Java packages (think namespace in .NET) it exports publicly, and the Java packages it imports or depends upon.  Anything that's public inside a bundle's packages is only made visible to other bundles in the OSGi environment if its explicitly exported by the bundle.  In this way, the problem described in my original post is solved.  The internal dependencies of a bundle are kept within the bundle and don't affect the bundle's public consumers.  I admit that I don't know .NET well enough to comment intelligently, but I'm willing to bet that this problem would be unsolvable currently in .NET (meaning .NET 4.0).

I had a look at semver.org, and generally agree.  Maven interprets versions semantically (like semver) only when they abide by the major.minor.revision[.qualifier] format, where major, minor and revision are nonnegative numbers and the optional qualifier is any string.  If no qualifier is present, semver rules apply.  If a qualifier is present, the qualifier, which can be any string, is interpreted according to its string ordering.  If the version does not abide by this format, then its string ordering is used.  Maven provides for the specification of which version precedes another via a separate mechanism in this case if need be, and I believe it works with the version range syntax described earlier in this post.  The best practice, however, is to abide by the major.minor.revision[.qualifier] format.  Prerelease bits are often handled by a "-SNAPSHOT" suffix and is a separate discussion.  Given what follows in this post, I don't think you need to worry about it in NuGet.

I posted http://social.msdn.microsoft.com/Forums/en/clr/thread/ee41510c-1e54-4245-8ec0-33af3c597a3a on the MSDN forums before I saw semver (but after years of maven usage), and I still think that it holds.  With semver's recommendation that you append text qualifiers to the end of versions to indicate prerelease bits, it becomes incompatible with .NET's requirement that each element among the dotted four of a version be a 16-bit unsigned integer.  This introduces impedence between semver's and .NET's versioning strategies.  My proposal, as given in the MSDN post, provides the same information as text qualifiers, but lives within the limitations imposed by .NET.  Here it is in a nutshell, quoted from the post (see the post itself for examples):


<major>.<minor>.<revision>.<stage><iteration><buildNumber>

The elements for major & minor are what you would expect.  A "revision" is a bugfix-only release; no new features.

Stage is a single-digit number, with the following conventions:  1 => alpha, 2 => beta, 3 => milestone, 4 => release candidate, 5 => release.

Iteration is a single digit, allowing for up to nine releases of alpha, beta, milestone, release candidate & release; in practice, I've never seen this number greater than 3.  Further, if the stage is 1-4, then iteration is 1-9:  a one-based index of the iteration.  If the stage is 5 (meaning release), then the iteration is always zero -- it's a release, after all.  Nothing follows a release with the same version.  It's the end of the road for that <major>.<minor>.<revision> version.  The next thing to follow a release would be a new version, greater than whatever the last release is.

Lastly, buildNumber is the index of the build during that iteration, and it's three digits long (remember, each of the numbers between the dots must fit into 16 unsigned bits).  This allows for up to 999 (if one-based) or 1000 (if zero-based) builds for that iteration and stage.


There is one thing that I would say about version ranges in maven:  they're almost never used.  Almost every project you'll see out there in maven-land will specify exact versions for dependencies.  The reason being is primarily for predictable and repeatable builds.  The old adage "If it ain't broke, don't fix it!" comes to mind.  Your NuGet repository, IMHO, must insist that once a package is published to it, it can never be changed.  This is in line with the whole repeatable build thing.

I hope this helps, and I highly recommend a review of Maven.  Many of the problems that you're encountering have been solved in Java land.  You actually have it a little better in .NET, thanks its formal specification of assemblies, complete with mandatory versions!

-matthew

Coordinator
Nov 4, 2010 at 4:05 PM

I should clarify #2. In terms of looking at the version of a package, yes 3 == 3.0 == 3.0.0

In terms of defining a dependency, we're considering them not necessarily equivalent.

For example, if I have the following dependencies:

<dependency id="foo" version="3" />

We're considering that to mean that you depend on any 3.* version of Foo. So Foo 3.5 would match.

If you wanted more precision, you could specify more precision.

<dependency id="foo" version="3.0.0" />

Would mean you depend on Foo v3.0.0.0 or 3.0.0.1 or 3.0.0.* but not 3.0.1.0.

Does that make sense? The idea we're considering is when you specify a version (and not a range), we use the precision you give us to determine how precise our dependency matching should be.

Coordinator
Nov 4, 2010 at 4:10 PM

Hi Matthew, regarding #4, we've been having discussions with the CLR team to look at what support we may need in the core framework to better support these types of things. As you point out, some of the issues you point out are core to the DLR.

Also, we have to be mindful that NuGet packages aren't only .NET assemblies. For example, we have JavaScript only packages such as jQuery and jQuery Validation. Since those are built on a dynamic language, version ranges actually make more sense there. :)

As for SemVer, there's a couple of aspects of it. 1. In part, it's a recommendation for how to version your package. So we don't necessarily need framework support other than we might want to have our own version type that supports specifying the string at the end. For assemblies within a package, they obviously have use the AssemblyVersion attribute and will not be able to fully be SemVer, but can adopt some of the SemVer conventions such as how to version a minor update vs a breaking change update.

Nov 4, 2010 at 4:31 PM

I suppose I should clarify as well.  In terms of version comparison, I agree that 3 compares the same as 3.0, which compares the same os 3.0.0, which compares the same as 3.0.0.0.

If I express a dependency on something with version "3", I'm torn.  I acknowledge the notion of precision.  If it were supported and interpreted to mean the range "[3.0.0.0,4.0.0.0)" (using Maven range notation here), I suppose that would work ok.  However, I would discourage that as a best practice when used during builds (which, I know, NuGet doesn't yet do).  It makes sense given the use case of a developer manually adding a reference to a VS project, which NuGet is precisely designed to do right now.  I guess it boils down to the principal of least surprise.  If I say I depend on version 3, is it surprising at all that I mean "[3.0.0.0,4.0.0.0)"?  Hard to say, for me.  If you reject the notion of precision and require an exact version, a version range, or a set of version ranges, then it becomes immediately obvious that you're depending on an exact version or one version among many.  It seems to me that it boils down to use case.  Developer using UI/console to add a project reference?  Sure, use the notion of precision.  Project author defining dependencies to be used in a build?  Surprising, IMHO.  Since the .csproj file doesn't allow you to specify version ranges of referenced assemblies (I think), then maybe NuGet's UI could, as a convenience for the user, search for versions in the range "[3.0.0.0,4.0.0.0)", but it definitely adds only the single version found.

If the build tool (msbuild, VS, TFS build server, etc) were enhanced to support the full build lifecycle, documented for Maven for clean ("Clean") & build ("Default") at http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference), and NuGet were to be configured as a project reference resolver, then you could support the idea of precision as you describe and issue a warning that depending on version ranges (when precision would produce multiple possible versions) is decidedly not a best practice.  In Maven's case, maven is the dependency resolver.  In the case of msbuild and/or VS, you would need to be able to specify the reference resolver or resolvers (chain?) so that NuGet could resolve dependencies (references) transitively.

How do we get MS to define the build lifecycle, beyond just supporting pre- and post-build steps?  Is there a way for NuGet to install itself as a reference resolver for any mainstream .NET build tool?  It seems like this is going to require a change from atop the mountain, unless I don't know TFS build definitions well enough yet... :)

-matthew

Coordinator
Nov 4, 2010 at 4:38 PM

Well part of the reason we were considering this precision syntax (and others correct me if I’m wrong) was that we didn’t have a way of expressing “[3.0,4.0)”. So we thought taking advantage of precision was a nice way to do that.

You have a good point about being explicit. If we adopted the Maven style syntax, we could easily require explicitness and forget the precision idea.

Thus:

version="3.0" would mean 3.0, 3.0.0, 3.0.0.0 and not 3.0.0.1.

And version="[3.0,4.0)" would mean 3.*.

David, Rob, etc. Would love to hear your thoughts. Should we consider this change?

Nov 4, 2010 at 4:47 PM
Edited Nov 4, 2010 at 4:48 PM
haacked wrote: As for SemVer, there's a couple of aspects of it. 1. In part, it's a recommendation for how to version your package. So we don't necessarily need framework support other than we might want to have our own version type that supports specifying the string at the end. For assemblies within a package, they obviously have use the AssemblyVersion attribute and will not be able to fully be SemVer, but can adopt some of the SemVer conventions such as how to version a minor update vs a breaking change update.

That's exactly what my <major>.<minor>.<revision>.<stage><iteration><buildNumber> convention attempts to do.  Unfortunately, that means that the "golden" release number for everything becomes something other than x.0.0.0, which some people find aesthetically displeasing.  I figure we're all mature adults, and can safely ignore the stuff from the third dot on, unless we really need to know.   So the "golden" release for marketing purposes is "1.0.0", but the actual AssemblyVersion, AssemblyFileVersion, etc., is "1.0.0.50000" (as per my convention with "5" meaning release).

The NuGet version, if it were separate, could look like what is becoming more and more frequent in Java versions, that is, "1.0.0.RELEASE".  Note that, in terms of string sorting, ALPHA < BETA < MILESTONE < RC < RELEASE.  I realize that things in the .NET world are a bit different (CTP, RTM, etc).  It works in .NET, too:  ALPHA < BETA < CTP < RC < RTM (Java's* MILESTONE == .NET's CTP, and Java's* RELEASE == .NET's RTM).  So, the following would all be equivalent, if my conventions were followed:

  • 1.0.0.50000
  • 1.0.0.RELEASE (Java-land)
  • 1.0.0.RTM (.NET-land)
  • 1.0.0 (Marketing version)

-matthew

* : To be fair, it's not so much a Java convention per se, as much as they are SpringSource conventions.  However, IMHO, SpringSource has their $#!+ together better than most and they are by far the most prevalent framework in use in Java.  I tend to listen when they speak.  And I used to work for them (but not on Spring.NET). :)

Nov 4, 2010 at 4:52 PM
Haacked wrote:

version="3.0" would mean 3.0, 3.0.0, 3.0.0.0 and not 3.0.0.1.

 

And version="[3.0,4.0)" would mean 3.*.

That example is enough for me to recommend dropping precision and preferring maven's range syntax.  Besides, I maintain that in build contexts, where I'm dying for you to go, you're probably not going to use version ranges much, if at all.

Nov 4, 2010 at 5:00 PM

You know, the more I think about it, the more I favor uniform .NET version numbers, avoiding semver prerelease text suffixes or a separate NuGet version string that allows nondigit characters.  My motivations are:

  1. This will be used almost exclusively in a .NET environment, where
  2. There are already several different version attributes around.  Further,
  3. If all the different versions (NuGet package, AssemblyVersion, AssemblyFileVersion, etc) are the same, then it's easier for me to think about.

The trade offs are:

  1. We would have to live with MS's 16-bit unsigned int limitation on each number, and
  2. We would have to know the conventions (1 => ALPHA, 2 => BETA, 3 => CTP, 4 => RC, 5 => RTM), that admittedly differ from MS's (major.minor.buildNumber.revision, see http://msdn.microsoft.com/en-us/library/system.reflection.assemblyversionattribute.aspx).

Microsoft's major.minor.buildNumber.revision never made sense to me, anyway, given semver's (and my own experience's) preference for major.minor.revision (where revision could also mean patch).

-matthew

Coordinator
Nov 4, 2010 at 5:09 PM

Keep in mind, that the package version can be different from an assembly version. Packages in NuGet can contain zero or more assemblies. So there’s not necessarily a one to one correlation.

Typically though, yes they will match up. But you could have the package version have more metadata than the assembly version in that case. For example:

MyAssembly v1.0.0.0 ß .NET assembly version.

MyAssemblyPackage v1.0.0.0beta

Nov 4, 2010 at 10:20 PM
Haacked wrote:

Keep in mind, that the package version can be different from an assembly version. Packages in NuGet can contain zero or more assemblies. So there’s not necessarily a one to one correlation.

 

Typically though, yes they will match up. But you could have the package version have more metadata than the assembly version in that case. For example:

 

MyAssembly v1.0.0.0 ß .NET assembly version.

MyAssemblyPackage v1.0.0.0beta

 

What examples could you give where a package contains multiple assemblies with differing versions?  The only reasonable example that has a package with multiple assemblies that I can think of off the top of my head is multiple assemblies with the same version but that target different .NET platform versions.  In the case of a package with a single assembly or my aforementioned example, I would expect, maybe even demand, that the NuGet package version be identical to the assembly's version.

Since I'm not too familar (yet) with packages that contain more than a single assembly version, I'd ask for some examples so that I can kick them around.

-matthew

Editor
Nov 4, 2010 at 10:23 PM
MvcScaffold has three assemblies as I recall, all different. Not uncommon.
>
Nov 4, 2010 at 11:12 PM
shanselman wrote:
MvcScaffold has three assemblies as I recall, all different. Not uncommon.
>

 Can you please point me to it so that I can have a look?  I saw http://aspnetmvcscaffold.codeplex.com/ but I wasn't sure that was the one you're referring to.  Thanks!

Editor
Nov 4, 2010 at 11:30 PM

Install-Package MvcScaffold

Nov 5, 2010 at 1:50 AM
Phil you may have already been saying this, but why not use the wildcard when you want precision? I think it's more easily understood than brackets and parentheses.

I go back and forth on the precision, but I could definitely adopt the idea that if you want to specify by precision, it's an explicit wildcard (*). That way it alleviates confusion and actually seems somewhat more intuitive. Thus like you said version="3.0" would draw out to the four octets, with zero for a lack of an octet. So like you mentioned 3, 3.0, 3.0.0 and 3.0.0.0 all mean 3.0.0.0.

And if you wanted any version of 3 you would use the wildcard i.e. version="3.*"

Hope this makes sense.
Nov 5, 2010 at 8:39 PM
ferventcoder wrote:
Phil you may have already been saying this, but why not use the wildcard when you want precision? I think it's more easily understood than brackets and parentheses.

I agree with your point on notational simplicity here, Rob, but caution you about the limitation that the wildcard syntax would introduce.  Consider the use case where a user wants to express something like "I'll take any 3.x.y.z version up to but not including version 4.0.0.0, except any 3.2 version, since they screwed the pooch on that series of releases."  The alternative syntax to the wildcard syntax in this example would be "[3.0.0.0,3.2.0.0),[3.3.0.0,4.0.0.0)".  Notice that the syntax should allow lists of ranges (as it does in Maven).  What would the wildcard syntax to support this example be?

Note that I realize that these are pretty miniscule issues when considering NuGet's current use within VS using either the GUI or console.  These become major issues when you start using these NuGet version expressions to drive your build's dependency resolution.  If you can consider and bake in a version syntax that supports using NuGet as your dependency resolver at build time, you've saved a potential backwards-compatibility issue when you do start using NuGet as your dependency resolver (note my jolly optimism here).

Having said all of this, I would still support using a wildcard syntax as a shorthand for the canonical range syntax because it does support a common use case.  The translation from wildcard syntax to canonical syntax is pretty straightforward.  Going the other way (from canonical to wildcard), however, is not (pending your response to my question above).

-matthew

Coordinator
Nov 5, 2010 at 9:06 PM

We just did an exhaustive search through the rubygems.org site, and I would say 99.9% of them only specify min version. At this point, I'm pretty convinced we should follow the Maven approach (or a subset thereof) for specifying version ranges http://maven.apache.org/enforcer/enforcer-rules/versionRanges.html. But we're going to have to implement binding redirects for assemblies that are not strongly typed.

While the interval notation might be foreign to some, if you dig deep, you'll remember learning it back in 6th grade (or 7th grade, or whatever). And the documentation for interval notation is free: http://en.wikipedia.org/wiki/Interval_(mathematics)

empty = all versions

(a,b) = {x | a < x < b}

[a,b) = {x | a  x < b}

(a,b] = {x | a < x  b}

[a,b] = {x | a  x  b}

(a,) = {x | a < x}

[a,) = {x | a ≤ x}

(,b) = {x | 0 < x < b }

(,b] = {x | 0 < x ≤ b }

[,b) = {x | 0 ≤ x < b }

[,b] = {x | 0 ≤ x ≤ b }

[a] = {x | x == a }

a = {x | a ≤ x }  // Note this is effectively a shorthand for [a,)

Note, we might punt on supporting multiple version sets in v1. We'll have to see if that adds more work or not.

Added bonus! Maven is Apache licensed, so we can port their interval parsing code from Java to C# and make use of their code, which means less work overall for us. In my book, score 1 for OSS! :)

Nov 5, 2010 at 9:23 PM
haacked wrote:

Added bonus! Maven is Apache licensed, so we can port their interval parsing code from Java to C# and make use of their code, which means less work overall for us. In my book, score 1 for OSS! :)

<joke>

Hell, then, why don't we just rename NuGet to Naven?  :)

</joke>

Nov 5, 2010 at 9:27 PM
Edited Nov 5, 2010 at 9:28 PM

<more-joke>

...aw, why don't we just port the whole damned thing?  Maven 3 => Naven 1?  J# won't be retired until 2015 anyway!  That's 4 years!  :)

</more-joke>

Nov 6, 2010 at 12:20 AM

Note that openwrap considers '3' to mean 3.*.*.* and will get whatever version is available that matches this.

We also have min and max versions that use this logic.

It doesn't make  much sense to have 3 mean 3.0.0.0, if you want to be precise you can be, if you're not precise then you're not, not being precise ending up being a precise definition is illogical.

Nov 6, 2010 at 12:43 AM
haacked wrote:

a = {x | a ≤ x }  // Note this is effectively a shorthand for [a,)Added bonus! Maven is Apache licensed, so we can port their interval parsing code from Java to C# and make use of their code, which means less work overall for us. In my book, score 1 for OSS! :)

and empty is shorthand for [0,) which coincidentally is also the emoticon for a frankenstein cyclops doing the dreamworks face

 

Nov 6, 2010 at 1:03 AM
serialseb wrote:

Note that openwrap considers '3' to mean 3.*.*.* and will get whatever version is available that matches this.

We also have min and max versions that use this logic.

It doesn't make  much sense to have 3 mean 3.0.0.0, if you want to be precise you can be, if you're not precise then you're not, not being precise ending up being a precise definition is illogical.

 Note that the current proposal is to have '3' mean '[3,)', i.e. 3 or higher, with no upper bound.  This matches the Maven semantic.  So you might just need to convert that into a min version in your nuget-openwrap adapter.

Nov 6, 2010 at 1:05 AM
loudej wrote:

and empty is shorthand for [0,) which coincidentally is also the emoticon for a frankenstein cyclops doing the dreamworks face 

Wait, which way are we tilting our head here? :)

Nov 6, 2010 at 3:38 AM
Matthew - that was not a considered case, so thanks for pointing that out. :D

Phil - True, Ruby is not a compiled language that actually looks at the version as part of the reference, so most of that stuff can be minVersion. .NET is a different animal and so implies slightly different rules.

Coordinator
Nov 6, 2010 at 3:41 AM

Understood. But consider Java is also a compiled language and follows this rule. I’ll let Lou chime in with the full reasoning. He’s got a lot of experience in this space that applies.

My summary of his argument is without making this the default, we’re creating a new sort of DLL hell. We’ve already seen it with things like several NHibernate packages with conflicting dependencies in our feed that in theory would work, but don’t because they don’t match up exactly.

By doing auto-binding redirects combined with min version, more scenarios will work. Not all of course, but those scenarios that would break under this new system would break under the old system. But with the new system, some scenarios that could work will now work, whereas by being strict, they always fail.

Nov 6, 2010 at 3:59 AM
Of course I'm throwing out the consideration that not everything is compatible and that is okay. I would rather have something that is guaranteed to work versus it may not work. In other words, I guess I prefer the stricter approach because I then know something is not quite right and I need to then do something.

We are not really creating a dll hell, it's already prevalent in .NET. The NHibernate example thrown out there is a real world scenario. In that case I have to think about what I would do manually to figure out how I could make them work or downgrade something until I get to a suitable usage scenario. I would rather have that be something that nuget supports at an explicit level, so at least I know I am doing it. That way there are no surprises when it doesn't work.

That said, there may not be a suitable usage scenario. In the world of dependencies, I want guarantees, not possibilities. If we go this route of possibilities, it means that sometimes people will get packages and they won't get their apps to run even with binding redirects due to some incompatibility between package versions. Then they will have to go get the specific version that .NET is complaining about. To me that kind of defeats the purpose.

Of course it could just be me.
____
Rob


Nov 6, 2010 at 6:27 AM
Edited Nov 6, 2010 at 6:28 AM

Rob, this was pretty much the way I felt until talking to Lou.  Well, I still partially feel that way, but it comes down to choosing the lesser evil.  We're dealing with two opposite types of DLL hells:

  1. Make the binding too strict, and things that are compatible refuse to run together
  2. Make the binding too loose, and things that are incompatible are allowed to run together, which can blow up

The old fashion windows DLL hell tends to be #2, while the .NET DLL hell is normally #1.  Doing the MinVer + BindingRedirects gets us closer to #2, but I think it solves more issues than it causes.

Note that we are targeting the following behavior.  Scenario #1:

  • A uses C v1
  • The feed has both C v1 and v2

Here, we'll use C v1.

Scenario #2:

  • A uses C v1 and B uses C v2.  Both are uses in a project
  • The feed C v1, v2 and v3

Here we'll use C v2.  So basically, we'll use the lowest version that matches the requirement, which limits the risk.  Apparently, Maven works that way, while Gems always gets the very latest (per Lou!).

Of course, in scenario #2, it's possible that C v2 is not 100% backward compatible with C v1.  But that in itself does not guarantee that things will break, because A may be using C is a way that's not affected by the breaking change.

It is certainly possible that using C v2 will break A, which obviously is the scenario causing concerns.  And that concern is legitimate, but all in all I now really feel that allowing things to try to run together (which requires testing), is better than preemptively blocking something that might have been fine.

If we don't allow this, it becomes very difficult to ever come up with newer versions of packages that may have some small breaking changes, because any package that tries to use it becomes instantly unusable with packages that use the older one.  And note that if a component goes through a major rearchitecture that breaks everything, the recommended approach is to use a new package ID altogether.

Nov 6, 2010 at 8:26 AM

There are also ways to have a guarantee about a family of related packages (like nhibernate, dynamicproxy, log4net) if you also have a metapackage for a stack or suite which coordinates them in known-to-work combination. That's how the "rails" gem itself works - you can up/downgrade its version to indirectly choose an exact combination of related subsystems in the rails stack. (It's one of the rare places you'll find a large number of =x.x.x versions in a gems).

In a past life we used log4net, nhibernate also uses log4net and dynamicproxy, monorail used log4net, dynamicproxy and nhibernate, etc. In orchard there are again many of the usual suspects.. Once the dependencies are a mesh, and they come from a variety of sources that update on different schedules (and frequencies!), your options boil down to:

(a) wait for the planets to align and upgrade when there is a moment where there are public bits that use the same versions for all overlapping resources
(b) grab the source code for each of the conflicts in your stack and build them locally
(c) throw in binding redirects - which has the same net effect as (b), really, unless you're willing to debug and repair version incompatabilities when rebuilding from source

I think Haack also has a really good point - there's nothing magic about Java that enables Mavin to use the "best" package when an "exact" one isn't possible. It's a managed, compiled, strongly typed, object oriented language - and the kind of things that will cause a runtime failure under a clr binding redirect would cause a runtime failure on that jvm. The only difference is (c) is the default behavior on the jvm, but the clr will "refuse to work in case it doesn't work". :)

Final thought - to David's point about finding the happy place between the two hells - I think that's exactly right I'm hopeful it's entirely possible. A big problem with DLL hell was the system32, and environment path, and com registry gave any changes to shared libraries a machine-wide impact. The really nice thing here is the assemblies we're talking about are scoped and version-locked within the boundaries of each particular application - which is a huge difference. Even if they are assemblies which are also present in the GAC, the binding redirect in the app will dictate which version of that assembly that app will use from that location. (Not saying the GAC is necessary - just that it's not harmful.)

 

Nov 6, 2010 at 11:37 AM

Let me jump in at this stage to lecture against the use of strongly-named dlls once more.

If you strongly-named, you jump into a resolving behavior by fusion that enforces you can only run against the exact version you compiled against. To fight this, you either never change the assembly version (and then why in the name of god sign anything to start with?), or you add long lists of assembly redirects (which moots the point of having versions to start with), and in that scenario you can't ever update dependencies at runtime or deploy time.

If versioning is taken care of at the package level, you should let your dependency manager deal with versions, not the Fusion loader. Having both is a big waste of time and duplication of functionality.

So no Lou, there is not only a, b or c. THere's also (d), where you don't sign and Fusion ignores version numbers completely and will take whatever it's given.

As for how to make sure packages are compatible, there's many known ways to test this stuff. We've got a lot of stuff around that happening in OpenWrap in the coming weeks.

Nov 6, 2010 at 2:41 PM
The problem being that people strongly name things just so that they're GAC-compatible, which seems to be a common customer request.
Coordinator
Nov 6, 2010 at 3:27 PM
Another reason, and perhaps even more prevalent reason, that customers strong name assemblies is so that their assemblies can be referenced by other strong named assemblies.

For example, if Log4Net wasn't strong name signed, then no strong named assembly could reference it. That would limit its use as a utility library.

Also, I don't think we can, nor should, prevent users of Nuget from strong name signing their assemblies should they choose to.

Nov 6, 2010 at 4:03 PM
serialseb wrote:

Let me jump in at this stage to lecture against the use of strongly-named dlls once more.

Remember that we're not writing assemblies and trying to decide whether to strong named them or not.  We're writing a package manager and we want it to work with what's out there today as much as possible.

So while having a discussion about the goodness/badness of strong naming is fine, when it comes down to it it is somewhat moot from NuGet's point of view.

In the end, I don't think we can avoid dealing with binding redirects to make things work together in some scenario.  Are you implying that you never use binding redirects in Open Wrap?  If so, I think you'll end up finding that it is too limiting given the libraries that are out there today.

Nov 6, 2010 at 11:13 PM
serialseb wrote:

If versioning is taken care of at the package level, you should let your dependency manager deal with versions, not the Fusion loader. Having both is a big waste of time and duplication of functionality. 

Beautifully said, a truly Zen moment of clarity. Merging three concerns - loader, version policy, and crypto policy - complicates each of them with the other's details.

But that's not about to change, and everything Brad, Phil, and David have said is exactly correct. The two asks you get whenever you share an assembly are for the official bits to "allow partially trusted callers" and to be signed. Ironically - those requests are rarely because of permission set or authenticity concerns - it's from people who want to (or have to) use medium trust or the GAC features.

 

Coordinator
Nov 6, 2010 at 11:26 PM
If I understand correctly, I believe the way Open Wrap solves this problem is to strip strong name signatures from assemblies in its packages. That's not an approach I'd feel comfortable with, modifying the bits the users intend to package.

However, we can and should take a long term view of the root problem we're trying to solve. If we want versioning taken care of at the package level and not by the Fusion loader, we should make a request of the CLR team for this ability so we can take advantage of it in the future.

Would someone want to take a stab at describing the desired functionality in the CLR?

Nov 7, 2010 at 6:26 AM

Does OW really creates a new binary that doesn't have the strong name?  Interesting approach, but the down side is that it'll break if some other binary that doesn't go through OW references the binary that has been modified.  With NuGet, one goal is that just because you use it to get one package does not necessarily mean that you have to be in a different NuGet world throughout.  Though I may very well misunderstand what OW is doing here.

To me the request to the CLR is simply to have a way to perform bind redirects at runtime without modifying the config file.  This can be achieved today via an AssemblyResolve event, but we've gone down that path and found that it is not currently usable.  We did start a CLR thread a while back discussing exactly that, but nothing concrete came out.  We could resume this talk.

Nov 10, 2010 at 12:35 PM

We warn strongly against using strong-naming when building a package, for all the reasons i've highlighted before.

There's a couple of ways we have to solve the strong-naming problem, either by enforcing a unique value for the version, which would let us bypass the CLR versioning, or by resigning assemblies whenever a package is installed using a post-install hook ,using whatever  key the consumer has. It wouldn't be much of a problem for pure IL assemblies, but mixed-mode is much more of an issue.

At the end of the day, if people *really* want to add strongly-named assemblies because they require side-by-side execution, then we won't block that (as I have some scenarios here where this is a desired feature), we just strongly discourage people.

As far as I am aware, you cannot use AssemblyResolve to override a strong-name into another, as the type names will still be enforced, making that method useless.

And indeed, we will not support in any way, shape or form assembly binding redirection in openwrap, as those only solve the narrow problem of dev experienc within VS, not the run-time composition that OpenWrap is also built to support. I would much rather spend time with code that unsigns on packaging and resign on install than support the xml stuff.

As for what I'd want from the CLR is to let me provide my own fusion implementation if I so wish. The CLR was designed specifically to not allow this (unlike our friends in the Java world), but maybe 10 years later the architects of the platform will be more encline to revisit the problem.

 

And David, no I don't believe it's a moot point to discuss for the future of any dependency manager. If you want to support all the nightmare scenarios and corner cases Microsoft has created for itself over the last 10 years, you'll end up building a monster of complexity that could be resolved by providing new direction and guidance.

Existing libraries do not play well with a pacakge-managed world, and the transitioning period should be towards an understood way of managing dependencies, not 25 because "that's the state of things", you can leave that to the existing platform rather than add it to a new one, don't you think?

Nov 12, 2010 at 4:15 PM
serialseb wrote:

And indeed, we will not support in any way, shape or form assembly binding redirection in openwrap, as those only solve the narrow problem of dev experienc within VS, not the run-time composition that OpenWrap is also built to support. I would much rather spend time with code that unsigns on packaging and resign on install than support the xml stuff.

Note that we are taking the binding redirect route in NuGet.  In the end, it's not fundamentally different from what you're doing with your resigning logic: it allows initially incompatible versions to bind together at runtime.  I prefer the binding redirect approach because it is less magical, and matches what the user would probably end up with if they were to do everything manually without NuGet.  And that's probably an important difference between NuGet and Open Wrap: we mostly aim to automate what the user would normally do by hand, but they still end up with more or less the same as they would have done by hand.  While in OW, you rely on fancier runtime behavior to make everything work, but your goal is not to match what the user would have done without OW.  Just a different approach to the problem! :)

And I agree that having some longer term CLR direction discussions is good (and we've been chatting with CLR a bit), but right now we have a very concrete short term goal of making life easier for .NET developers with the various libraries that they're using *today*.

Nov 15, 2010 at 8:57 PM

Hmm. We ask people to think about their versioning stragegy (including strong naming) *today* so they don't have to deal with the tightly-coupled .config approach they *had* to rely on because of limitations in the CLR fusion extensibility *yesterday*. Different way to reach the same aim, but we take the approach of teaching people about the solutions they can use rather than go for the "stroke people in their existing knowledge" way.

Long term against short term?

More constructively, how are you going to generate your assembly redirects? Automatically upgrade everything up to the version being imported?

Nov 15, 2010 at 10:18 PM
serialseb wrote:

More constructively, how are you going to generate your assembly redirects? Automatically upgrade everything up to the version being imported?

Essentially.  See my 'Nov 5 at 11:27 PM' post in this thread for some more details on the algorithm.

Nov 17, 2010 at 10:38 PM

Well, I was only referring to, once the resolving has decided what version gets selected, do you redirect 0-(v-1) to v where v is the version you selected, I assume it's a yes.

I fully expect different package managers to use different dependency resolution algorithms. We've just built a new exhaustive one, and the next step is to provide one that balances stability and quick resolving over extensive compatibility, as we do have to deal with system-wide dependencies, so the resolution algorithms themselves will always behave differently.