Why MSBuild for Package Restore?

Topics: General
Coordinator
Feb 18, 2013 at 8:51 PM
Here's a question... Why is it important that package restore can happen via msbuild? Why not just require nuget.exe install before you can build the project? Combine that with the ability to restore in VS via A) simple UI gesture, and B) PowerShell (perhaps a parameterless install-package command).

Other platforms seem to expect that after you clone a repository, you have to run an install command to get required packages before the project can work.

Are we just doing ourselves more harm than good by shoehorning package restore into MSBuild?
Coordinator
Feb 18, 2013 at 8:53 PM
Related: "Upcoming feature in NuGet 2.3: Auto import .targets and .props file from package"
http://nuget.codeplex.com/discussions/433564
Feb 18, 2013 at 9:04 PM
Visual Studio 2012 can run tests after build with a simple checkbox. How about adding such checkbox "Restore NuGet packages before build"?
Does Visual Studio have support for project specific settings yet?
Feb 19, 2013 at 7:49 AM
I agree with removing package restore from the .csproj, on the client the restore can be done through the addin. On the build server it is more robust to call package restore before building the projects.
Since package restore does not update the HintPath it is more reliable to call package restore before the build. This way a list of assembly folders can be selected from the package repository to pass into "ReferencePath".
Feb 19, 2013 at 10:39 AM
Perhaps that is the best solution. If it is universally known and accepted that performing a build does not perform package restore then everyone will be forced to solve it through tooling (built in VS and CI server features) and/or in their build scripts, e.g. nuget.exe install.

I will take a bit of adjustment for developers, but I think it's viable.
Feb 19, 2013 at 3:43 PM
Removing this feature would be quite rough, from the point of view of someone who doesn't have a use for the "Auto import .targets and .props file from package" feature [e.g. myself and my colleagues, with which we maintain a few dozens internal .NET libraries].
Feb 19, 2013 at 7:44 PM
Removing the MSBuild approach would not break anything if a valid alternative is offered, that's the beauty of the current MSBuild approach. Those would still just work, and people could migrate away from it whenever they feel like.
Coordinator
Feb 19, 2013 at 8:11 PM
So here's a related question...

How would you get the bits on your machine in the non-MSBuild approach? Other platforms either have the package manager built into the platform or it's a separate install. NuGet won't be a Windows component. Does that mean a NuGet install that gives you nuget.exe on your path?
Feb 19, 2013 at 8:33 PM
A NuGet installation on PATH is probably a good idea. As you say, this follows the pattern of RubyGems, NPM, etc.
  • Windows - publish a downloadable installer, plus a package on http://chocolatey.org/
  • Linux - add to relevant APT sources with dependencies on the relevant Mono packages
  • OSX - no idea ;-)
Feb 19, 2013 at 8:38 PM
Edited Feb 19, 2013 at 8:40 PM
The latest version of nuget.exe is directly available on https://nuget.org/nuget.exe.

Those who want older versions can fetch the NuGet.Commandline package from nuget.org through https://www.nuget.org/api/v2/package/NuGet.Commandline/{packageVersion}

There are tons of possibilities to have it installed on your system the way you want. PATH is a good idea.
Coordinator
Feb 20, 2013 at 12:34 AM
The core team met today and talked about Package Restore quite a bit. We reached a consensus within the team that we do want to explore a non-msbuild-based approach for package restore.

We will be doing more design work in this space; stay tuned. And if you have input into how you think it should be designed, feel free to add those thoughts here in the meantime.
Feb 20, 2013 at 12:46 AM
I could get you a Nuget installer built for Windows in no time. An installer package and a "plain old .zip file" (for those people that check their tools into their source control) would be quite sufficient. It works well for us in the WiX toolset. I'll let others debate the hardship of moving to this model over MSBuild.
Feb 20, 2013 at 5:59 AM
Does .NET ClickOnce allow the updating of PATH?

ClickOnce seems to work well for the NuGet Package Explorer. I'm not sure if it's even possible for a console app though.

Obviously it's trivial to build and publish an MSI. It's also quite trivial to simply download NuGet.exe and it to your path manually (I did this a long time ago).

In addition to the above, my favourite would be a Chocolatey package. One of the reasons that the separate install approach works so well on other platforms is because of APT. It's trivial to type sudo apt-get install nuget. It's even easier to type cinst nuget. Ideally we'd have an APT equivalent built into Windows but in the meantime Chocolatey is the closest thing we have.
Coordinator
Feb 20, 2013 at 6:11 AM
While I too like Chocolatey, it is another barrier. I am not comfortable saying, "To restore packages, you need to install NuGet. To install NuGet, you need to install Chocolatey."
Feb 20, 2013 at 6:21 AM
I agree it should not be the only method of installation but it would be great if it was provided alongside the other options for those of us 'in the know' ;-)
Coordinator
Feb 20, 2013 at 6:24 AM
Yeah, that's a good point. Makes sense.
Feb 20, 2013 at 8:00 AM
Package restore is a 'bootstrap' action to setup the build environment. It should be executed prior to the actual build. Currently, all project files execute the bootstrap action during their project builds which, as we know, not always works well.

Perhaps this is a workable alternative (apart from having nuget.exe in your path):
  • When enabling 'Nuget package restore' for a solution, not only copy nuget.exe etc. in the generated .nuget directory, but also a bootstrap.csproj file
  • This bootstrap.csproj file knows how to restore nuget packages for the solution
  • The bootstrap project is included in the .sln file in such a way that msbuild will buld this bootstrap project first.
In this scenario, building a solution will start with the bootstrap phase which will restore all nuget packages. When this step is complete, the remaining projects have their packages setup correctly and can be built without a problem. Observe that there is no longer a dependency on nuget package restore in the individual project files. The only drawback I can see is that package restore doesn't get activated for when building individual projects. But I think this is justifiable by explaining that package restore is a bootstrap action and bootstrap actions cannot be part of a concrete build step.
Feb 20, 2013 at 12:18 PM
adamralph wrote:
In addition to the above, my favourite would be a Chocolatey package.
Just asked @ferventcoder:
cinst nuget.commandline ftw (nuget.org is one of chocolatey's default sources).
Feb 20, 2013 at 12:24 PM
Xavier beat me to it. While it would be nice to see nuget on chocolatey.org proper (which we might see a copy from the official chocolatey account in the future thanks to myget services), all pkgs on nuget.org are installable with choco. To qualify for PATH they simply need to include an executable in the pkg.

--
____
Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder
Developer
Feb 20, 2013 at 2:32 PM
Totally agree with MdeJ
Coordinator
Feb 20, 2013 at 6:22 PM
The bootstrap project is interesting as it does enable the 'msbuild foo.sln' approach to still work, without having to run a nuget command first.

The potential downsides I see are:
  1. It doesn't address the ASP.NET Website scenario where there is no solution file or msbuild (when using WebMatrix primarily)
  2. Each solution then needs to have an extra bootstrap project in it. But perhaps we put the .nuget folder contents into that project instead of a solution folder, offsetting it.
I also have the question: Is this okay for Mono and XBuild?
Feb 20, 2013 at 8:40 PM
@jeffhandley it's great to see you taking an active interest in Mono and XBuild in such decisions.

I've managed to make some of my OSS projects fully cross platform compatible with CI builds for both. I'd love to answer the Mono and XBuild question definitively but unfortunately I can't since I don't use package restore at all. (Forgive me, but I only see downsides to using package restore, no upsides.)

Intuitively, I'd say if package restore works today for Mono and XBuild in those cases where it works for MSBuild (which I think it should for Mono 2.10.9 or higher) then I see no reason that the bootstrap project shouldn't be a viable solution for Mono and XBuild.

If the building of the bootstrap project before all other projects is somehow forced to happen with no required changes to the internals of MSBuild, then I would guess no changes would be required in XBuild either.

Do we have any idea how to ensure that executing msbuild Foo.sln always builds the bootstrap project first?
Developer
Feb 20, 2013 at 8:43 PM
There's a little-known trick to hook into pre-build event of a solution.
Developer
Feb 20, 2013 at 8:44 PM
We don't really need the boostrap to be a project. It can just be a .targets file as it is now.
Feb 21, 2013 at 12:00 AM
We need to make sure that the steps to restore packages outside of msbuild are straightforward. The problem is that in a large solution, you end up with one packages.config file for each csproj, so the steps to locate them all are not trivial.

Specifically, I'm thinking about CI scenario, and in particular the logic with have in Kudu (Azure git support) to build projects. Currently, all we have to do is locate the relevant csproj and run msbuild on it, and all the magic happens. We don't even need to know what other projects it depends on, since msbuild takes care of that.

But if we first need to recursively search for all the relevant packages.config to run nuget.exe on them, it gets complicated.

And one more thing: how can we even know where projects expect the Packages folder to be without doing some crazy parsing on the csproj files?

Just trying to think through issues... :)
Developer
Feb 21, 2013 at 12:35 AM
RE: "But if we first need to recursively search for all the relevant packages.config to run nuget.exe on them, it gets complicated. "

You don't need to recursively search for relevant packages.config. You run the bootstrap .targets which restores all packages for the whole solution.

I think no matter what solution we come up with, there'll be tradeoff for one scenario or another.
Coordinator
Feb 21, 2013 at 12:56 AM
Here are my takeaways at this point:
  1. We need to get out of the business of modifying every project file, but we must ensure the scenario of restoring packages and then building that project is easy.
  2. At most, we need to have a single, simple command to run to restore packages for an entire solution.
  3. We can still consider an msbuild-based approach that takes us from one command to zero.
  4. We can consider an installer around nuget.exe (in addition to a good chocolatey install experience) to get nuget.exe on the machine, but this should not be required.
  5. Whichever approach we take needs to work on Mono with Xbuild.
  6. As a bonus, it would work with ASP.NET Websites created through WebMatrix as well.

Machine-Level Install

I've been unhappy about the contents of the .nuget folder needing to be updated by hand, in clumsy ways. And I also don't like having to check a heavyweight nuget.exe into my .nuget folder every time there's a new version of NuGet. Those gripes make me want a machine- or user-level install of the package restore artifacts so that I don't have to carry them in every project. For that, I'm willing to require that install for those that want to work on my repositories.

I could see a balance on this where:
  1. If there is a solution-level pre-build targets file in place, that means package restore is enabled.
  2. That targets file is flexible and lets me either check nuget.exe in with my solution or use nuget.exe from a known path (environment variable).
Feb 21, 2013 at 5:32 AM
Note that the default NuGet.targets already has support for downloading nuget.exe on the fly (using WebClient). This is a decent 3rd alternative to having nuget.exe preinstalled or committed. It's kind of the best of both worlds.
Feb 21, 2013 at 12:24 PM
Edited Feb 21, 2013 at 12:27 PM
May I point out that 'nuget install', when run from the command line, does not run the install scripts for the packages (e.g. Install.ps1).

What this means is that if you attempt to do "package restore" by hand, the packages don't actually get installed properly.

I would be happy to have a restore.bat file for my solution which pulls down all the packages (especially those containing .targets and .props files which get copied out of the packages folder during install) using nuget.exe except currently this wont work due to aforementioned limitation.
Feb 21, 2013 at 12:37 PM

From: jtu100

May I point out that 'nuget install', when run from the command line, does not run the install scripts for the packages (e.g. Install.ps1).

What this means is that if you attempt to do "package restore" by hand, the packages don't actually get installed properly.

Package RESTORE is not a reinstall, and it doesn't need to be. When you install a pkg in VS, it does all the install work then and then you check in your changes. Let me ask a question here, when other team members install packages and then you get latest, are you ALSO required to install the package(s)?

The package restore only removes the necessity of committing pkg files to source control, not all of the work they've already done.

--
____
Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder
Feb 21, 2013 at 12:53 PM
Idea:
  • "nuget install" should be renamed "nuget restore"
  • "nuget install" should eventually disappear, or be replaced by a command which is able to update csproj, run ps1 scripts and so on.
Feb 21, 2013 at 1:06 PM
Nuget.exe provides a different workflow. If it is aliased as restore, I'm okay with that. But IMHO it should not go away.



--
____
Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder
Feb 21, 2013 at 1:24 PM
Edited Feb 21, 2013 at 1:30 PM
ferventcoder wrote:
Package RESTORE is not a reinstall, and it doesn't need to be. When you install a pkg in VS, it does all the install work then and then you check in your changes. Let me ask a question here, when other team members install packages and then you get latest, are you ALSO required to install the package(s)? The package restore only removes the necessity of committing pkg files to source control, not all of the work they've already done.
Please excuse me for thinking install means install :P

The problem I face is that some packages install assemblies that are used by the build process. e.g. MSBuildTasks.

But it appears these cannot work with package restore because they must be present before msbuild kicks off. Hence they must be checked, which is obviously not ideal.

Is there a way to solve this 'build integrated' packages problem?
Feb 21, 2013 at 1:35 PM
jtu100 wrote:
Is there a way to solve this 'build integrated' packages problem?
That's one of the goals of this discussion :)
As Jeff posted at the top, this issue also relates to this one: http://nuget.codeplex.com/discussions/433564

Calling pkg restore before msbuild kicks in could be one approach, another one that has been suggested is to hook into the solution's pre-build step.
Feb 21, 2013 at 4:14 PM
While we're talking about package restore issues, I'd like to bring up this other thread about project files that are used in multiple solutions: http://nuget.codeplex.com/discussions/431666.

The real issue is that references (and any imported targets/props files), have a path relative to the csproj file, but package restore only has a single location relative to the sln file.

We need to have multiple packages dirs. relative to the projects and it should restore the right packages to the right package directories.
Feb 22, 2013 at 10:58 AM
We need to have multiple packages dirs. relative to the projects and it should restore the right packages to the right package directories.
Nope. Get your project structure in order.
Feb 22, 2013 at 10:58 AM
broggeri wrote:
Idea:
  • "nuget install" should be renamed "nuget restore"
  • "nuget install" should eventually disappear, or be replaced by a command which is able to update csproj, run ps1 scripts and so on.
I really like this idea
Feb 22, 2013 at 11:58 AM
lennybacon wrote:
We need to have multiple packages dirs. relative to the projects and it should restore the right packages to the right package directories.
Nope. Get your project structure in order.
If you read the other thread, I explain the project structure. There's multiple teams working in multiple areas in source control. A flat solution structure simply won't work.
Feb 22, 2013 at 8:47 PM
I would recommend to place each of what you call area into its own repository. Having it's own packages directory. Accepting that each area is a component like a one like a product you got from external.

The thing you call master - if being the executable - can reference all others by getting them as packages. If it's not your build server should take care of "build-all-projects" in the right order.

what do you think?