Locally-Installed Packages

Mar 16, 2012 at 11:39 AM

Hello,

We at SharpCrafters would like to discuss the possibility to implement and contribute a feature into NuGet. We understand that it would have high impact on NuGet, so before going into code we would like to have some good discussion and agreement about whether and how it should be implemented.

Support for locally-installed packages

A local package is a package that has been installed by a third-party setup program (such as the PostSharp setup program, for instance) on the local machine. Local packages are installed in expanded (unzipped) form, respecting the canonical NuGet directory structure. That is, local packages are not delivered as nupkg files.

When a local package is added to a project, no nupkg file needs to be downloaded. The local package is not copied to the solution-wide repository. Instead, the project refers to the local directory where the local package is installed (perhaps using a registry key as a level of indirection, as allowed by MSBuild).

Locally-installed packages will have precedence over other (nupkg) packages. When a dependency can be resolved by a local package, this local package should be used, unless specified otherwise by the user (by checking an option or answering a message box).

Motivation

Although NuGet is a great way to resolve dependencies between software components, a fair number of software tools are still better deployed using an old good setup program. Indeed, software tools are not only composed of APIes, but also contain user interfaces or build-time components. Deploying them to source control is not always feasible or desirable. However, software tools also contain APIes and it is desirable to use NuGet to resolve dependencies between these APIes. Currently, software tools vendors have to choose between NuGet and local installation, but there is no way to publish to NuGet any component installed locally by a classic setup program.

We are facing the issue with PostSharp (http://www.nuget.org/profiles/SharpCrafters). PostSharp Toolkits are ready-made aspects that integrate with popular frameworks such as, currently, log4net or NLog, and later Enterprise Library and other. NuGet is the dream way to distribute toolkits because of dependencies. But without support of the feature we are proposing, our toolkits are not compatible with locally-installed PostSharp. Instead, PostSharp must be re-installed from NuGet for each solution. Local installation still the preferred way to install PostSharp (especially in the Getting Started experience, which is critical to all software vendors), so we see this feature as critical for us.

We would be willing to spend time and/or money on this feature, but would first like to discuss it in details.

We're looking foward for your feedback!

Thanks.

-gael

Mar 16, 2012 at 1:32 PM
Have you heard of chocolatey? http://chocolatey.org

This will probably meet 90% of what you are looking for, the rest could be overcome with pull requests.
____
Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder

Mar 23, 2012 at 8:06 AM

Hi Rob,

Thank you for your reply and sorry for late answer.

Chocolatey is cool, but I don't think it addresses the same issue. We are not looking for another way to deploy our software (we have 3, already). We would like the local setup experience to integrate with NuGet. This is very different. We really want to keep our installer. We just also want NuGet to recognize packages that we installed locally using the installer, instead of trying to pull them from the network and duplicate them.

-gael

Mar 23, 2012 at 2:16 PM

Ha! Keep your installer. I'm not sure you understood what chocolatey does. If you look at a package like notepad++, it downloads the native installer and executes it silently.
Basically there are three methods used in chocolatey - use native installers, use powershell to do the work, or just drop an executable you want on the path in chocolatey.

But what you described in that last email is exactly why chocolatey exists. If I use chocolatey to install git extensions, it is going to ensure I have msysgit also installed. If it is already installed it moves on to git extensions. And the the upgrade story is really where the wins begin.

Most chocolatey packages are tiny, because they have only a powershell file in them with a one liner that says here is the url, optional 64bit url, and here are the silent switches (and other installer switches I want to pass).

It's not an additional distribution method, it's an enhancement to your current distribution method.

It's really your choice in the end though. I'm just inviting you to explore chocolatey and see if it will work for your needs. It is basically nuget for applications/tools/machine wide configuration.

Mar 23, 2012 at 2:19 PM

Also, the recognizing already locally installed items outside of choco is on the list as a future enhancement. Chocolatey is open source on and we accept pull requests.

Apr 24, 2012 at 9:23 AM

I wonder if I could have the attention of one of the owners regarding this feature request. I have prototyped an implementation available on http://nuget.codeplex.com/SourceControl/network/forks/gfraiteur/NuGet, but there are still several days of work before it will be production-ready. Before going forward, I would like to know what the project owners think. 

Thank you.

-gael

Apr 24, 2012 at 2:15 PM

In the upcoming 1.8, we did add a similar feature to NuGet. We allow a repository to contain unzipped packages in place of .nupkg files. However, it's still different from your proposal in that we still copy the package binaries to your solution folder, and it is only used in the preinstall packages from templates. There's no support for main stream use (yet).

I think your idea has the potential. We'll need to discuss more within the team first. I haven't looked at your fork yet, but I have a couple of simple questions for you:

  • How does user specify a locally-installed repository in VS? (btw, I don't think we should call this feature "locally-installed packages" because we already have the concept of local packages (packages on your hard drive.)
  • How are the assembly reference paths specified in the .proj file?
  • This would mean all users are required to install PostSharp before they can compile the project. This really goes again the nuget's principle and it breaks the package restore scenario. What is your take on this?
Apr 24, 2012 at 2:38 PM

I didn't look at your code, but I have also been thinking about this kind of feature, where nothing at all needs to be copied into the solution.

The tricky point is @dotnetjunky's second point: what does the reference look like in the project file? Normally, it's a relative path to the assembly (e.g. ..\packages\EntityFramework.4.2.0.0\lib\net40\EntityFramework.dll). But clearly, with the local install, the path cannot be relative, nor can it be absolute (since the local folder won't always be the same). So it would need to be based on some environment.

Can you describe how this works in your case?

Apr 24, 2012 at 2:46 PM

Thank you for your answer.

I wrote a more detailed specification, you can download it from http://dl.dropbox.com/u/9884935/NuGet%20Local%20Pavkages%20-%20RFC.docx.

I noticed that LocalPackage was already taken, so in code it's named RegistryPackage.

1. RegistryPackage are specified in registry, typically added by the setup program (see documentation and the fork).

2. Assembly references don't have a hint path. The setup program must configure HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\AssemblyFolders. I think it's consistent with the way SDKs are deployed today. An alternative would be to add a level of indirection in the MSBuild file (a registry value lookup), although it may be unsupported by DTE (especially for web sites project systems).

3. You are referring to a use case where a team has already selected a package and wants to use in a set of projects. This feature targets the "getting started" experience, when a single individual somehow found our product, and decided to give it a try. From this moment, we want to provide the easiest experience possible, which is: to download some installer program and execute it. We are planning to build a feature  (an "aspect gallery", allowing to add behaviors to code just with the mouse) on the top of NuGet, and we would like this feature to work in both scenarios: local setup, and NuGet. Given that our NuGet packages count only for a fraction of our total downloads, we can't just push people to NuGet. I think we would loose some business by doing so. There are other good reasons to prefer local setup for individual developers and small teams, one of which is performance (thanks to NGen).

I think Package Restore should be updated so that it checks if "Registry Packages" are installed locally, and fails if not. That would be fine to have an "upgrade path" (from local setup to network nuget), but I don't think it's a must-have.

Apr 24, 2012 at 6:37 PM

We'll spend some time reading your specs and get back to you.

Apr 24, 2012 at 6:57 PM

Thanks for the extra info. This is a little bit different from the kind of feature I had in mind, and probably solves a different problem. What I had in mind was more something like a local cache of expanded packages that can be used directly from the cache location. This is in contrast with our current cache, which avoids package downloads, but doesn't otherwise affect how the package is handled in the project.

Example workflow:

  • I clone some project Foo from github, and it uses packages A and B (both containing assemblies)
  • I happen to have package A in my 'expanded cache', but not package B
  • When I build, the system is smart enough to reference A.dll directly from my 'expanded cache', while B.dll is treated the standard nuget way

The important point here is that the author of the Foo project doesn't need to know which package will end up coming from where when I install it. i.e. a given package is not tied to being always one way.

Note that this is just something I was thinking about, and I'm not clear what the implementation would be when it comes to the reference in the csproj.

So anyway, I didn't mean to randomize your feature with something that maybe is not similar enough to be discussed together. :)

One potential issue with your approach (if I understand correctly) is that you are requiring any environment that needs to build your code to have the 'registry package' installed. Isn't that going to be a tough requirement on general purpose CI servers where you can't easily install things?

Apr 25, 2012 at 1:30 PM

I understand the issue with my approach. On the other side, most of component vendors today provide installers as the only way to distribute their software. Take DevExpress, for instance. If you want to try their product, you have to download their huge installer, and it is going to install all kinds of things, including code samples, demo apps, VS-integrated help, toolbox items, and so on. Typically, such components can't be packages as NuGet. Now, what if a NuGet package has a dependency on some DX component? My feature request addresses specifically this scenario. Of course, you can replace DevExpress in my example by Visual Studio SDK or Microsoft CRM SDK. How would you like these SDKs to be deployed? The answer I'm proposing is to deploy them with an installer, as usually, but install references to them using NuGet.

I thing my RFC could be extended so that the NuGet repository could know that a package is available only as installed software, and would simply display a message box asking to visit the vendor's web site (ferventcoder/Chocolatey could come to rescue at this point).

For deployment on build servers, we already propose the two approaches (Nuget/VCSor local installation). As said before, our objective with this feature is to improve the Getting Started experience (read: improve conversion rate of trials into customers). When a customer is sold, I suppose he would do the additional effort to move to a "proper" component deployment model. But think of VS SDK: how is it typically deployed on a CI server?

-gael