Support Global Package Installations?

Topics: General
Coordinator
Jan 18, 2013 at 5:58 AM

When installing a package from nuget.org, files go through the following flow:

  1. nupkg file is downloaded from nuget.org and saved in a cache (max of 100 packages cached) here:
    %LocalAppData%\NuGet\Cache
     
  2. nupkg file is copied into the solution packages folder:
    D:\git\jeffhandley\MyAwesomeSolution\packages
     
  3. When the project is built, binaries from the package are copied into the project’s bin folder:
    D:\git\jeffhandley\MyAwesomeSolution\CoolProject\bin

The 3rd party is of course standard build behavior that is not specific to NuGet.

Every new solution that references a package results in a new copy of the NuGet package, plus new copies of the binaries from the package, as applicable for the project’s target framework.  Ignoring the DLL copies into the bin folder, the result is N+1 copies of a NuGet package where N represents the number of solutions referencing the package.

Room for Improvement

NuGet has room for improvement here.  Ideally, the N+1 result would be reduced down to 1, but I believe reducing down to 2 is more attainable, but implementing Global Package Installs.  Other package management systems support Global package installations, and NuGet should too.  We could leverage Global package installations in the following manner:

When a user is installing a package from nuget.org (or any feed), they could choose to specify the “Global” parameter.  For instance:

  • C:\> nuget.exe install Microsoft.AspNet.SignalR -global
  • PS> Install-Package Microsoft.AspNet.SignalR -Global
  • The NuGet package manager dialog could have an option for global installations too

The result would be similarly altered:

  1. nupkg file is downloaded from nuget.org and saved in a cache (max of 100 packages cached) here: %LocalAppData%\NuGet\Cache
  2. nupkg file is copied into the Global packages folder: %LocalAppData%\NuGet\Global
  3. When the project is built, binaries from the package are copied into the project’s bin folder: D:\git\jeffhandley\MyAwesomeSolution\CoolProject\bin

Again, with the 3rd being standard build behavior.

What It Would Take?

In order for Global package installations to work, NuGet must change how binaries from packages are referenced.  Instead of having A) packages.config, which lists the packages installed into a project, and B) <Reference> nodes in the project, with hint paths pointing to the solution packages folder, we would instead need to inject a <NuGetPackageReference> node into the project file, and have build-time resolution of the binaries that need to be referenced.

Once build-time reference resolution was in place, then the <NuGetPackageReference> handling could look in:

  1. The solution-local packages folder (or the repositoryPath configuration specified in nuget.config)
  2. The global packages folder

Thoughts?

I'd like to hear others' thoughts on this feature.  We've heard some complaints about how many times packages like Microsoft.AspNet.Mvc are sitting on your machines and about how many times we have to copy files around.  Would global package installations alleviate those problems for you?  Would it solve other issues?  Would it cause you problems?

Jan 18, 2013 at 6:14 AM

So in essence, this is a "NuGet GAC"? I'm not sure that in these times of cheap disk space N+1 copies of a package on your system are actually a big issue. If I look at some projects on my machine, there are only a couple of packages that are actually used in more than one project.

Some related thoughts:

  • What with packages like jQuery that do not have assemblies? I still will have multiple copies of the .js files on my system.
  • What with, say, MvcScaffolding? If I were to install that one globally, I lose a lot of functionality of that package (namely changing T4 templates)
Coordinator
Jan 18, 2013 at 6:23 AM

For content packages like jQuery, you would still have multiple copies of the .js file on your system, just like the DLLs that get copied into your bin folder.  You just wouldn't have the *package* folder copied with every solution.

For packages that have binaries that support lots of frameworks, you presently have the binaries for every framework supported sitting in each solution's packages folder, regardless of which frameworks the solution is using.  With a global package installation, you'd have 1 global package install, and then each project would copy into its bin the assemblies that apply to it.

For MvcScaffolding, where it is expected that you can edit files in the package's folder, you could simply choose to not install that package globally.

Jan 18, 2013 at 6:40 AM

Would that also mean you might end up in a situation where:

A. You install a package which happens to be no longer available in your local cache (100 packages limit)

B. You have that very same package already installed as a Global package for another solution (maybe months ago)

C. NuGet will try to fetch the package from the cache and the repository without looking in the Global packages folder?

I'm trying to think about what advantages there would be in maintaining a Cache folder in addition to the Global packages folder. Seems to me the Global packages folder would be a great cache location as well? Thoughts?

Coordinator
Jan 18, 2013 at 5:18 PM

The distinction between cache and global will be an interesting aspect.  I expect cache would remain in between installation operations and web requests, but I wouldn't expect to treat the Global packages folder as a secondary cache.  In principle, it seems wrong to conflate the two for me.

As far as why you'd want a separate cache from global, I see a few benefits:

  1. If you install a package local, but then decide to install it global, that global install can come from cache.
  2. Without a cache being separate from global installs, it's harder to track what's truly installed global.
  3. With the two combined, you'd no longer be able to easily garbage collect/clear the cache.

I propose we'd keep the two completely separate, with the belief this would result in a much cleaner implementation.

Jan 18, 2013 at 5:23 PM

In my reality, I use quite limited number of solutions with much bigger number of projects. So my calculation is:

n - solution count

m - project count per solution

worse case is:

1 [in cache] + n [in packages folder] + (n * m) [in bin folder]

after elimination of packages folder: 

1 [in cache] + (n * m) [in bin folder]

Alsmost the same. My vote not to bother at least with proposed solution. 

Developer
Jan 18, 2013 at 5:24 PM

This seems like a great feature to me.  Would this mean creating projects with lots of NuGet packages in the template (ie MVC apps) would be quicker because it wouldn't have to copy so many files?

Also, switching to <NugetPackageReference> nodes or something similar instead of direct references in the project would have a lot of other benefits, so please try to do that in any case. :)

Coordinator
Jan 18, 2013 at 5:42 PM

@dsplaisted VS Templates is one area that could benefit from global packages, yes.  VS templates could have their packages installed globally (perhaps with an opt-out mechanism).  That would make File->New Project A LOT faster!

Developer
Jan 18, 2013 at 6:16 PM
Edited Jan 18, 2013 at 6:16 PM

Jeff,

I think there's an opportunity to get rid of the cache and just use the Global for everything. If we are going to address the disk space issue, why not going all the way? The Global can act as the new cache. To address your concerns:

  • If you install a package local, but then decide to install it global, that global install can come from cache.

    The first time you install a package local, we still put it in global instead of the cache. It's the same amount of work that we are doing today.

  • Without a cache being separate from global installs, it's harder to track what's truly installed global.

    It can easily be done by putting some marker file in the package folder in global. 

  • With the two combined, you'd no longer be able to easily garbage collect/clear the cache.

    With the marker file above, we can easily clear the packages that are not being referenced.

  • Jan 18, 2013 at 6:23 PM
    Edited Jan 18, 2013 at 6:24 PM

    It seems like a <NugetPackageReference> might help solve the Extension SDK problem if it can dynamically resolve references based on solution/build configuration.

    The main issue is that ARM and x86 assemblies need to be provided for WP8 and Store apps that use any native code. If you could provide both in the package and have NuGet use the right one based on what's being built, that'd rock.

    Granted, there are other things in Extension SDKs (like having reference vs. redist and debug/runtime folders), but that could all be accommodated by the NugetPackageReference resolution logic.

    Coordinator
    Jan 18, 2013 at 6:37 PM

    @dotnetjunky True, it could be done and while I'm open to this discussion on it, we should consider the purposes of the 2 stores separately and then while designing it decide whether 1 physical store can serve the 2 virtual stores easily without introducing potential issues.

    @onovotny Yes, I don't want to conflate features too much, but I doo think <NuGetPackageReference> would help us achieve other goals like you mentioned.  You could cross-compile your project against multiple frameworks much more easily, and you'd not need to reinstall packages after retargeting.

    Jan 18, 2013 at 6:47 PM
    jeffhandley wrote:

    2. nupkg file is copied into the Global packages folder: %LocalAppData%\NuGet\Global

    @jeffhandley Maybe I misunderstood this part: I understood it as the .nupkg being copied as is, maybe you meant the package contents are extracted into that location? In that case, forget my earlier question :)

    If not, here's why I was asking:

    jeffhandley wrote:

    ...I wouldn't expect to treat the Global packages folder as a secondary cache...

    I didn't mean to cache (write) any packages in there in between installation requests and web requests. I meant using the Global packages location as a fallback package source (read).

    You mentioned some valid reasons to keep cache and global separate. Makes perfect sense to me. What's interesting to me is the scenario where you want to install a package locally when it has already been installed globally into another project while it's no longer available in the local cache.

    Would you look into the global packages folder for the package or not? In essence, would NuGet attempt to fallback upon the global packages folder as a local package source in case it can't find the package in the cache? I think this would be a really handy feature in the event the main package source is unavailable or when working disconnected: NuGet would have a best-effort local package resolution mechanism built-in to fallback upon.

    Coordinator
    Jan 18, 2013 at 6:55 PM

    @xavierdecoster If there's a desire for the global package installs to be used as a cache, then that backs up @dotnetjunky's proposal to merge the two.  I would expect them to serve different purposes, but I could very well be the minority here.

    I am hoping to get feedback on the *general* idea of supporting "Global Installs" for packages though, without getting too mired in design details.  Do you think the concept of global installs is worthy of investment?  Is that a good direction for NuGet to go in?

    Jan 18, 2013 at 7:06 PM

    +1 for the general idea. I've seen this request before, don't see any harm in supporting it as an opt-in feature.

    @jeffhandley Noticed @dotnetjunky's proposal after posting my comment and the page refresh. Yep, guess we're thinking in the same direction :)

    Developer
    Jan 18, 2013 at 7:30 PM

    It's important to note that Global Install is not completely related to build-time package resolution. Global Install basically _requires_ build-time resolution, but we can do build-time resolution WITHOUT Global Install.

    I think build-time resolution is a _really_ Good Thing(TM), and I like Global Install to, but it's important to note you don't necessarily need one to get the other :).

    Jan 18, 2013 at 7:37 PM

    You already have this, but poorly implemented, and with no easy management UI.
    It is hierarchical nuget.configs. if you want a global install location, config it that way.

    Well, that is, once you fix the current implementation.

    As an asside, disableSourceControlIntegration should be supported this way too.

    Coordinator
    Jan 18, 2013 at 7:42 PM

    @EdditGarmon Good point that you can pull off a global package installation folder today.  But it's an all-or-nothing approach and doesn't let you have some packages installed globally and other packages installed locally.  More importantly though, it still results in Hint Paths in your references that point to that global install folder, and that is hard for teams to use unless they demand a certain path.  Even if we invested in the combination of hierarchical config + repositoryPath, it would still fall apart pretty easily I think.

    Jan 18, 2013 at 7:50 PM

    Another question, how would this new build time resolution be projected for plugins like ReSharper and CodeRush?

     

    Jan 18, 2013 at 10:21 PM

    What problem is this supposed to solve? The drives which I store my code on are not about to run out of space (far from it) so I don't care about having a copy of a given package version in multiple solutions. Will this proposal deliver any benefit to me?

    Jan 19, 2013 at 12:13 AM

    This is exactly how Ruby Gems work. For any given Gem, it is only ever downloaded exactly one time. The Gem Path is relative to your Ruby installation, so while you might not always install Ruby in the same place, file lookup is always guaranteed to work. For example, oauth-0.4.7 will only ever be downloaded one time. If you create 1000 different projects that use oauth-0.4.7, you have exactly one copy of that Gem on your system, all pointing to the same place.

    The same thing would be true in the .NET space. NUnit-2.5.10 would only ever be downloaded one time. Even if you have hundreds of solutions on your machine, you have one copy of NUnit-2.5.10. It's location would be a known location. Just make it C:\NuGet\Packages\<package> and move on with your life. Not everyone needs to configure all the things.

    I pointed this out back at Issue #1995 - http://nuget.codeplex.com/workitem/1995

    Jan 19, 2013 at 12:52 PM

    @jarrettmeyer I'm not sure whether your comment was in reply to mine or not but if it is, I still don't see any benefit to me. Currently my local git clones are 3.5GB in total (peanuts) so disk space saving is of no importance to me. Perhaps my example is a little low, but even the next two orders of magnitude (35GB/350GB) will fit onto a typical modern SSD (350GB would be an *enormous* amount of projects).

    Whilst the fact that this is how Ruby Gems works is interesting, I don't see what benefit it would bring to NuGet. Regarding your comment about configuration, I don't see how this would help. I still have to install the package into a project if I want that project to use it. NuGet already takes care of the management of that package on disk so why do I care where exactly that is and whether it's duplicated?

    More generally, the reason I am questioning this proposal is to ascertain a level of priority. I'm not against it but I do think it has very little value if any. The core team have already said they are not able to work on all the items in the backlog which are of value to the community (as per recent conversations about marking issues as 'UpForGrabs'). This is understandable as they only have a certain amount of resources to work with. I'd much rather see the core team concentrate on higher value proposals before addressing this one.

    Feb 8, 2013 at 5:56 PM
    Global packages would be a great thing for Native libraries. Native libraries (like Boost, OpenCV etc.) are huge and contain headers, sources and binaries (both build-time libs and runtime DLLs) and symbols (PDBs) for multiple architectures. This runs into gigabytes in some cases. Having all of this replicated for every solution on the machine sounds like a huge waste. Combining the power of distribution that NuGet provides along with the disk savings that global install provides would make this a very compelling scenario for developer building and consuming C++ Libraries.