Using NuGet without committing packages

Mar 14, 2011 at 7:14 PM

Last week, I blogged about how you can avoid commit packages by restoring them at build time using nuget.exe. This addresses one of the most voted issues on our list (this one).

Someone posted an interesting comment on my blog about a current limitation:

This is an interesting approach but I am having trouble applying it to my environment because I have two source feeds. I use a local feed for some internal libraries, and the default NuGet fedd for the remainder. I can see how to specify the source repository on the command line, but this only allows one source repository at a time.

Do you have a suggestion on the right approach to restore packages that aren't committed when multiple source feeds are required?

 

So the problem is that nuget.exe can only use one feed at a time, which breaks this workflow. We're trying to figure out the best way to address this.

One option is to have nuget.exe piggyback on the same list of feeds registered in the VS UI. While this makes sense, it may not work well for users who want to run this on a CI machine. For those scenarios, it probably makes more sense to have the relevant list of feed be carried along the state of the app (i.e. committed in some way).

One option is to have an app level config file that would list the relevant feeds. Another is to allow multiple feed URL's to be passed to nuget.exe.

Let's start a discussion on what people think would work best for the various scenarios where this feature can be used (i.e. dev box vs CI machine).

Mar 16, 2011 at 11:49 AM

Hi,

On my CI machine, I want to be able to control which feeds are available to use, so that I get a build of known quality, from components where I have some idea where they came from.  I don't want the build to grab arbitrary packages from all over the internet depending on where the developer got them in the first place! 

At the same time, I like the idea of having a list of feeds in an app-level config file so I could check where they came from, and as a dev, I can always re-create the app on my local machine with this information (unless the package has been pulled, obviously). 

So I wonder if a good approach would be to keep the app level config file, with a switch on nuget.exe where I could override where the file is pulled from if required.

Cheers for a great feature!

Miles

Mar 16, 2011 at 7:19 PM

When you say "where I could override where the file is pulled from", are you referring to the config file (i.e. using custom config file that's not in the app) or to the packages (i.e. specify custom feed URLs)? Just making sure we don't have a misunderstanding :)

Mar 16, 2011 at 11:26 PM

Sorry, I wasn't clear - I think I meant the second, i.e.

 + record in packages.config the additional hints about which feed a package came from;

 + include a switch on nuget.exe to control whether those hints are used to locate packages, or whether they are ignored and packages are downloaded from a set of specified feeds;

As an illustration, I suppose I'm thinking of something like:

1. # nuget.exe install packages.conf -o ../Packages

2. # nuget.exe install packages.conf -o ../Packages -f mySpecificFeeds.conf

Example 1. would look in packages.conf for the feed hints and the package list, whereas 2 would ignore feed hints in packages.conf completely and only use the feed list from mySpecificFeeds.conf

Not sure it's too dissimilar to passing multiple feed urls, except that there's more opportunity for the feed information to live outside the app/build file.

Mar 16, 2011 at 11:58 PM

Yes, this generally makes sense. We've been thinking about the idea of keeping track of the feed where each package comes from in the packages.config. And passing additional feeds through the cmd line is also an option. Note that technically those don't have to completely replace the ones in packages.config, but they can simply appear first in the search list, such that if they contain a package it will always come from there rather than the other one.

Mar 17, 2011 at 12:55 AM

This would solve my enterprise use case...

One other potential scenerio... developers pull from the public feed, but the build server has a local cache to reduce download times...ect.

sent from my mobile

On Mar 16, 2011 5:58 PM, "davidebbo" <notifications@codeplex.com> wrote:
> From: davidebbo
>
> Yes, this generally makes sense. We've been thinking about the idea of keeping track of the feed where each package comes from in the packages.config. And passing additional feeds through the cmd line is also an option. Note that technically those don't have to completely replace the ones in packages.config, but they can simply appear first in the search list, such that if they contain a package it will always come from there rather than the other one.
>
>
Mar 17, 2011 at 1:12 AM

As long as all your packages are in the same place, you can make this work today by specifying your local cache as the source of the 'nuget install' command. It's the multi-source scenario that's not yet supported.

Mar 17, 2011 at 8:04 PM

but they can simply appear first in the search list, such that if they contain a package it will always come from there rather than the other one.

I think this highlights the difference I was trying to get across - on the CI server, I want it to error if the packages are not in the list of feeds that I provided, rather than falling back to the package hint.  The hints could mean packages are downloaded from anywhere on the internet, which would then mean the build was not controlled.

Mar 17, 2011 at 8:37 PM

I see. We can certainly look into having the explicitly passed in feeds be exclusive.

Mar 21, 2011 at 8:58 PM

"As long as all your packages are in the same place, you can make this work today by specifying your local cache as the source of the 'nuget install' command."

How does one do that through the command line?

Mar 22, 2011 at 12:30 AM
devadept wrote:

"As long as all your packages are in the same place, you can make this work today by specifying your local cache as the source of the 'nuget install' command."

How does one do that through the command line?


Nevermind, I got it working. My pre-build event command line looks like:

nuget install $(ProjectDir)packages.config -Source C:\MyLocalLibrary -o $(SolutionDir)packages

Mar 22, 2011 at 1:21 AM

Yep, that's the way to do it!

Mar 27, 2011 at 4:21 AM

Any updates in terms of being able to call "nuget install packages.config" and specify which source should be used?

Apr 6, 2011 at 11:21 PM
Edited Apr 6, 2011 at 11:32 PM

I like where this discussion is going!  There is one other problem that has occurred to me -- what to do about updated or removed packages?

Let's lifecycle this by one step:  Suppose that we get this multiple-feed source thing working, I use it to deploy version 1 of my library, and all is good.  (what comes next is from the standpoint of keeping a developer's workstation up to date, and might not apply to a build server -- I'll note why as I go).

So now I decide that I don't need that package anymore, so on my dev machine, I use nuget to uninstall it.  Now, everything is great on my machine -- the unwanted assemblies are gone from my lib folder, any uninstall ps1 files ran, etc.  But what happens when the guy down the hall from me does a get latest and then a build (triggering the nuget.exe as discussed above).  Is there a way for it to determine that it needs to uninstall the package? Will his machine be "littered"?

(you can apply the same use case, I think, if I want to move my project from using version 1 of a package to version 2)

With a build server,  this  is probably a non-issue -- you start with a clean directory on every build.  But with the development workstations, I don't want the penalty of deleting my lib folder and copying all the packages down from the network every time I do a build -- I want to take advantage of the "only copy if changed" feature, but I also want to make sure that packages that are not used get uninstalled.  Is there any reasonable way to do that?  (maybe a locally maintained packages file that nuget.exe can compare with the source-controlled one?  kind of like how Visual Studio keeps a local project settings file that doesn't get checked in -- the .csproj.user file)?

EDIT

You know, I just realize, you probably don't want the uninstall to run on the second workstation -- it might interact with a project in a way that you do not want it too -- trying to remove references that are already removed, etc.  Maybe it's good enough if each packages contents are in a separate directory.

Apr 6, 2011 at 11:36 PM

@jmarsch: note that 'uninstall' can mean two different things:

  1. Full package uninstall, including removing assembly references and files from the project. Only one dev needs to do that, as they will then commit the updated csproj file
  2. Deletion of the package's folder

#1 is not a problem, so I think your focus is on #2, in the scenario where you don't check in those folders. Right now, there is no support for it, but it would be conceivable to have a flavor of the command that looks at a set of packages.config files, and not only gets all those packages, but also deletes any existing packages that are not in the combined list.

Please open a bug if you think that's an interesting scenario.

Apr 7, 2011 at 5:34 AM
Edited Apr 7, 2011 at 5:36 AM

G'day guys,

I have just created two separate forks that in combination will satisfy your needs.

 

http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetMultipleSources

This fork allows you to specify multiple package sources via the command line.  You simply semi-colon delimit them. For example:

nuget install packages\packages.config -o packages -s \\dev-server\packages;https://go.microsoft.com/fwlink/?LinkID=206669

 

http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetRepositoriesConfigSupport

This fork allows you to specify a path to the "repositories.config" file (this a file that NuGet creates to keep track of the paths to all "packages.config" files within a solution).  Therefore the following command:

nuget install packages\repositories.config -o packages

Is equivalent to:

nuget install Project1\packages.config -o packages

nuget install Project2\packages.config -o packages

...

nuget install ProjectX\packages.config -o packages

 

We are now using both of these features combined to negate the need to check in all packages.  Now we simply must commit the "repositories.config", all the "packages.config" files and then perform an initial seed of dependencies by calling:

nuget install packages\repositories.config -o packages -s \\dev-server\packages;https://go.microsoft.com/fwlink/?LinkID=206669

 

I hope this can be of use.

Apr 7, 2011 at 3:18 PM
@dant199 That's great! I was looking for that functionality!

Just wondering.... do u have a merged version? How do u suggest to combine them?

On Wed, Apr 6, 2011 at 11:34 PM, dant199 <notifications@codeplex.com> wrote:
> From: dant199
>
> G'day guys,
>
> I have just created two separate forks that in combination will satisfy your
> needs.
>
>
>
> http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetMultipleSources
>
> This fork allows you to specify multiple package sources via the command
> line.  You simple semi-colon delimit them. For example:
>
> nuget install packages\packages.config -o packages -s
> \\dev-server\packages;https://go.microsoft.com/fwlink/?LinkID=206669
>
>
>
> http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetRepositoriesConfigSupport
>
> This fork allows you to specify a path to the "repositories.config" file
> (this file contains a list of all "packages.config" files within a
> solution).  Therefore the following command:
>
> nuget install packages\repositories.config -o packages
>
> Is equivalent to:
>
> nuget install Project1\packages.config -o packages
>
> nuget install Project2\packages.config -o packages
>
> ...
>
> nuget install ProjectX\packages.config -o packages
>
>
>
> We are now using both of these features combined to negate the need to check
> in all packages.  Now we simply must commit the "repositories.config", all
> the "packages.config" files and then perform an initial seed of dependencies
> by calling:
>
> nuget install packages\repositories.config -o packages -s
> \\dev-server\packages;https://go.microsoft.com/fwlink/?LinkID=206669
>
>
>
> I hope this can be of use.
>
> Read the full discussion online.
>
> To add a post to this discussion, reply to this email
> ([email removed])
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this discussion on
> CodePlex. You can unsubscribe or change your settings on codePlex.com.
>
> Please note: Images and attachments will be removed from emails. Any posts
> to this discussion will also be available online at codeplex.com
Apr 7, 2011 at 9:56 PM

Thanks Dan! I haven't looked at the code, but the features do make sense. About the second one, I think we might be getting rid of repositories.config at some point (David Fowler may have more to say about this). Possible alternatives:

  • Locate the various projects by looking at the sln file
  • Recursively look for all packages.config recursively
Apr 8, 2011 at 1:38 AM

No worries!  It's my pleasure to contribute, I hope to continue to do so.

@davidebbo
I suspected that may have been the case RE deprecating the repositories.config, but it seemed the most elegant approach to take right now.  Both features combined amount to little more than 10 odd LOC so backing them out/enhancing/replacing them would be trivial.

@abarylko
I have created a new fork that combines both features:
http://nuget.codeplex.com/SourceControl/network/Forks/dant199/NuGetDanTurner

Simply clone https://hg01.codeplex.com/forks/dant199/nugetdanturner using tortoisehg, open the solution and build :)

Apr 8, 2011 at 6:58 PM

It's actually fine to rely on repositories.config for now. When/If we move away from it, we can simply evolve this change alongside.

May 5, 2011 at 3:55 AM

Are these changes going to make it in to NuGet?

May 6, 2011 at 11:11 AM
Edited May 6, 2011 at 12:52 PM

Hi, we are really interested in these changes, to take the packages folder out of the SVN, as it causes some conflicts when nuget updates a package with a new version, because cannot delete the old folder and we need to do it by hand and commit the deletion to SVN

The changes suggested by @dant199 seem rigth to me and quite usefull. Any plans to get them into the official Nuget.exe release?

Added: at least, can anyone make a nuget.exe release with Dan's fork to download? Not familiar with tortoisehg/Mercurial, as we are using subversion... Thanks!

Thanks,
Sergi

May 6, 2011 at 6:08 PM

Finally, installed tortoisehg, cloned and compiled it, and it worked like a charm! great job, Dan.

Now I have two versions of nuget.exe in the tools folder, and use each other depending on the context.

Just for curiosity: the official nuget.exe comes as a standalone file, but my compiled version from Dan's fork needs nuget.core.dll. Is this correct?

May 6, 2011 at 7:33 PM

@sgisbert: the official one is ILMerged in order to just be one file. Note that the build process produces this merged file, so you should be able to get the one from Dan as one file too (it ends up under bin\NuGet\Console).

May 6, 2011 at 11:49 PM
davidebbo wrote:

@sgisbert: the official one is ILMerged in order to just be one file. Note that the build process produces this merged file, so you should be able to get the one from Dan as one file too (it ends up under bin\NuGet\Console).

@davidebbo: Oh! I didn't know this was possible. But I haven't been able to find the path to this merged file:

- I build the Solution from VS with no errors, but couldn't find the folder bin\NuGet\Console.

- I ran the build.cmd from the command line, but it doesn't build and raises lots of Code Analysis errors (3 in debug, 34 in release), so cannot find the file either. Do I need something special installed in order to run build.cmd?

Anyway, it's not a big issue for me: just copied that dll to the tools folder and checked it in to svn. It was more for the curiosity about seeing this merge in action.

Thanks!
Sergi

May 6, 2011 at 11:54 PM

It's normal that it doesn't get built from VS, but it should build using build.cmd. Might want to start a new threads with details on the errors you're getting. It could be some of Dan's changes that raise some code analysis issues that don't exist in the main build?

May 7, 2011 at 12:04 AM

These are the 3 errors I get running "build.cmd" (debug or release):

Build FAILED.
        "D:\Virtua\Desarrollo\NuGet\NugetDanTurner\Build\Build.proj" (default target) (1) ->       
"D:\Virtua\Desarrollo\NuGet\NugetDanTurner\NuGet.sln" (Build target) (2) ->
"D:\Virtua\Desarrollo\NuGet\NugetDanTurner\src\Core\Core.csproj" (default target) (3) ->       
(RunCodeAnalysis target) ->         
MSBUILD : error : CA1709 : Microsoft.Naming : Correct the casing of 'Nu' in namespace name 'NuGet' by changing it to 'NU'. [D:\Virtua\Desarrollo\NuGet\NugetDanTurner\src\Core\Core.csproj]  
MSBUILD : error : CA1709 : Microsoft.Naming : Correct the casing of 'Nu' in namespace name 'NuGet.Runtime' by changing it to 'NU'. [D:\Virtua\Desarrollo\NuGet\NugetDanTurner\src\Core\Core.csproj]
MSBUILD : error : CA1709 : Microsoft.Naming : Correct the casing of 'Nu' in assembly name 'NuGet.Core.dll' by changing it to 'NU'. [D:\Virtua\Desarrollo\NuGet\NugetDanTurner\src\Core\Core.csproj]

    0 Warning(s)   
    3 Error(s)

Developer
May 7, 2011 at 12:09 AM

Those are just fx cop errors that need to be fixed.

May 13, 2011 at 9:16 PM

Any word on this? +1 for this. is there a feature I can vote for?

Coordinator
May 13, 2011 at 11:41 PM

The issue to vote up is here.

http://nuget.codeplex.com/workitem/165

I’ll have to double check, but I think we’re a little behind on this feature, which might mean it gets moved to the next release.


Phil