Package from a Project (.csproj) file

Dec 10, 2010 at 3:49 PM

So I've been tinkering with NuGet to add some features I want for using this as a packaging tool internally where I work.

What I have right now is an extension in the Manifest to specify csproj files to "add" to the package.

For example:

<projects> <project file="TestProject\TestProject.csproj" configuration="Debug" platform="AnyCPU" /> </projects>

What this will do is look up the PropertyGroup for the combination of Debug and AnyCPU, and add <OutputPath>\<AssemblyName>.dll to lib/<RuntimeVersion>/<AssemblyName>.dll

But I've realized I really need is a way to say "This package is from this project.  So copy the assembly to lib/, but also use the assembly name as <id />, and the assembly version as <version />"

Adding this seems like a bit bigger of a feature and might not align with what the project is envisioning.  What I like about it, is the nuspec file can be "static" after you write it since, it pulls all relevant info from the project file / assembly info.

So how do people feel about that idea?  Or is there a prescribed way to accomplish what I want with this feature?

Dec 10, 2010 at 7:10 PM

Grabbing as much data from the project instead of having it in the .nuspec definitely makes sense.  In the extreme case, it should be possible to not even have a manifest at all, and build a package by just feeding on a csproj file.  Assuming that all the assembly level attributes in AssemblyInfo.cs have been nicely filled up, you wouldn't be missing much.  Also, you could automatically figure out the packages it depends, assuming they were installed into the project using NuGet.

Dec 10, 2010 at 9:52 PM

Cool I try to have something to show by Monday / Tuesday about what I was thinking!  I appreciate the feedback.

Dec 10, 2010 at 10:07 PM

Note that AnglicanGeek prototyped some ideas along those lines a while back, though I'm not sure exactly what he did.

Note that the way I see this working is that it wouldn't change the code or the nuspec format.  Instead, it would make the 'nuget pack' command smarter, such that you can write:

NuGet pack foo.csproj

Or maybe optionally pass both a .csproj and a .nuspec so it can use data from both, since not everything can be expressed in the project (e.g. there is no attrib for ProjectUrl)

The code would then just create a PackageBuilder (which the pack command already does) and fill it up with all the relevant info from the .csproj and/or the nuspec.

Or something along those lines! :)

Dec 10, 2010 at 10:18 PM

Yeah that would definitely be more portable, just extending pack.

Also I want to make sure I expose this correctly so you can do it programmatically through the API as well.

Dec 10, 2010 at 10:55 PM

Yes, there are definitely some services that currently only exist in the exe and that should be moved to a library that the exe calls into.  This would be one of them.

Dec 17, 2010 at 9:02 PM
Edited Dec 17, 2010 at 9:03 PM

I've created a fork here : http://nuget.codeplex.com/SourceControl/network/Forks/ecoffey/PackageFromProject of my changes.

It's rough and not ready to be merged in, but I'd like to expose it to some feedbacks and explain the ideas I had.

I started by trying to make my changes in PackageBuilder, or somehow decorate PackageBuilder and then just "override" it with my project file specific work.

But I didn't feel happy with that solution.  I realized PackageBuilder had a lot of hard coded assumptions and was doing a lot of work:

* It was responsible for reading from a Manifest and populating itself with those settings

* It was responsible for determining what IPackageFiles got included in the package

* And finally it was responsible for writing out the package and the manifest

I didn't want to have to populate all the files, do all that directory reading just to toss it out and replace it with the computed IPackageFile from the csproj file.

So I did some slight refactoring, mainly just pulling apart PackageBuilder into some separate classes and reimplementing it (as IPackageBuilder2 / PackageBuilder2), which resulted in the following artifacts:

IPackageMetadataProvider: this interface is responsible for returning a IPackageMetadata when GetMetadata() is called.

IPackageFileStrategy: this interface returns a IEnumerable<IPackageFile> when asked.

IPackageWriter / PackageWriter: one method, void Save(Stream stream) and is pretty much a copy / paste of the package creation that was happening PackageBuilder.

PackageMetadataMerging: a extension method on IPackageMetadata that merges two metadatas together, giving preference to the right operand

CompositePackageMetadata: implements IPackageMetadata and takes a params IPackageMetadata, merging them all together.

CsProjectFileProvider: implements IPackageMetadataProvider and IPackageFileStrategy.  Returns computed metadata, and a single (wrapped in a set) PhysicalPackageFile pointing to the built binary.

NuSpecProvider: implements IPackageMetadataProvider and IPackageFileStrategy.  This is really just doing the work in PackageBuilder for populating itself from a Manifest.  Uses a few IPackageFileStrategy implementations to keep the same "what files to add to the package" semantics that PackageBuilder has.

You can look in PackCommand to see how these all come together to support the default of building from a nuspec file, but also the added functionality of merging metadatas and writing a set of files.

Some Concerns:

* Not sure if ManifestMetadata is a good "vanilla" IPackageMetadata implementation to be using where I return instances of IPackageMetadata

* FileVersionInfo vs Assembly.ReflectionOnlyLoad : You can see this SO question for more info (http://stackoverflow.com/questions/4466019/fileversioninfo-and-assemblyinfo), but essentially I'm not sure what the "right" way is going to be to get at the AssemblyVersion attribute in the project assembly.  Cecil (http://www.mono-project.com/Cecil) might help with this issue, but I'm not sure about the project stance on adding new dependencies.

 

Phew that was a lot of typing!  Please give me feedback and lets see if we can't get this officially merged!  Also, is there a better way to continue this discussion?  Mailing list, IRC, etc?  These forums seem a bit clunky for all of it.

P.S. I can fix formatting at merge time, or if someone can provide me with a solution specific ReSharper override.  I don't feel like rejiggering VS / ReSharper to match the non-MS formatting style :-P

Coordinator
Dec 17, 2010 at 9:49 PM

It would be a lot easier to review if you submitted it as a code review to our reviewboard server: http://nuget.codeplex.com/wikipage?title=Code%20Reviews

Can you attach a build of the vsix from your fork to the discussion so we can try it out easily?


Phil

Dec 17, 2010 at 10:00 PM

Cool I'll submit a review request up there momentarily.

 

How do I attach a file to this discussion?  Also I was only messing with the command line client, so I'm not sure how it's going to work in VS yet.

Dec 17, 2010 at 10:10 PM

I followed the reviewboard instructions, but when I run hg postreview -l and give it my password I get:

abort: you are not logged in (103)

 

Seems like reviewboard.nupack.com is taking a long time to process a login (through the web ui); maybe this command is just not waiting long enough before failing?

Dec 20, 2010 at 3:42 PM

I get the same error trying hg postreview -l today.

I logged into reviewboard.nupack.com and tried to manually create a review request.

I generated the diff with:

hg diff -c 638

And when I click 'Create Review Request' I get this back:

The file 'Build/Build.proj' (rc5d1d3a3b89a) could not be found in the repository

Not really sure what I should be doing here.

Developer
Dec 20, 2010 at 4:37 PM

I haven't tried updating my plugin but right now I am synced to 421caf934997 (in the hg postreview repository) and it works for me. Try that revision and see if it works.

Developer
Dec 20, 2010 at 4:49 PM

Ok I've pulled the latest changes and have been seeing all of those errors that people have been complaining about. It looks like the authors of the extensions are making changes to support reviewboard 2.0 and have maybe broken something for our 1.5 RC version (though I'm not sure). If you update to a32439ae1dc6 then it should just work. I'll update the code review instructions for now.

Hope this helps
David 

Dec 20, 2010 at 5:01 PM

How do I update to a32439ae1dc6?  TortoiseHg Repository Explorer doesn't show that as a Revision Set.

Developer
Dec 20, 2010 at 5:13 PM

Try going to the command line and typing hg up a32439ae1dc6. It's the change before "Added support for ReviewBoard API 2.0".

Dec 20, 2010 at 5:19 PM

Ahhh you meant in the reviewboard hg extension repo, sorry :-)

Ok, so how do I make a patch from a particular commit?  Since I've done a hg pull from nuget thinking that would help.  If I submit a review request now you won't actually get my code :-P

Developer
Dec 20, 2010 at 5:33 PM

What does hg postreview -l do for you now?

Dec 20, 2010 at 5:37 PM

I was able to feed it the rev of my merge.  I have the review created, I'm just filling in the description and will publish it shortly

Dec 20, 2010 at 5:43 PM

Created here : http://reviewboard.nupack.com/r/202/

Sorry that was such a headache to get in.  Also sorry the diff is so noisy because it's a merge :-/

Like I said still a hg / reviewboard neophyte, let me know what I can do to make this an easier process!

Developer
Dec 20, 2010 at 5:53 PM

Sorry to ask you to do this but can you not merge and try sending the diff again?

If you do merge, make sure you follow these steps:
hg pull
hg update {main's changeset}
hg merge

The key here is to update to main before merging that way the diff will only contain your changes.

 

Dec 20, 2010 at 6:00 PM

Sure, what's the best way to pull this history out?

If you look at http://nuget.codeplex.com/SourceControl/network/Forks/ecoffey/PackageFromProject I have one commit which is pretty much all the work, and then the merge that tweaks my changes to mesh with some of the movement that had taken place.

 

Also in your example from above what is "main's changeset"?

Developer
Dec 20, 2010 at 8:15 PM

Ok I just pulled your repository. Are you familiar with mercurial? You'll need to create a new fork to fix everything. After you have your new fork, What you can do is strip the 2 merges you did (using the the mq extension http://mercurial.selenic.com/wiki/Strip). After stripping the merges take your changes you can make sure you are updated to main's changes (your working directory should be synced to main's latest changes in this case dffc5570085b). After doing that you merge with your changes (hg merge). Then resubmit your review.

Dec 20, 2010 at 9:07 PM

This is the first time I've used mercurial, but I am familiar, roughly, with the concepts coming from git.  Just not super familiar with the lexicon yet, and the appropriate workflow, so my apologies for causing you a headache.

 

My fork has been deleted and recreated here: http://nuget.codeplex.com/SourceControl/network/Forks/ecoffey/PackageFromProject

And here is the new review with a much saner looking patch set : http://reviewboard.nupack.com/r/206/

I didn't create it with the --longdiff option since that was reaching wwwwaaaaayyy back into history, because of my earlier commit.  I think this will be at least easier to review and see what I did, and then we can worry about my idiocy with regards to mercurial :-P

Also let me know how my fork looks; there are two heads floating there that I haven't merged yet since I wan't sure what the Right answer was.

Developer
Dec 21, 2010 at 1:26 AM

That's good, don't merge until you're done making the review feedback. 

Dec 21, 2010 at 2:17 AM

I see a lot of discussions about the fork, but I couldn't find a description of the new feature that was added.  Could you briefly describe it?  e.g. what does a sample cmd line look like, what input does it take and what does it produce?

Dec 21, 2010 at 3:15 AM

It's pretty rough right now, but at the moment with the command line nuget tol you can do:

 

nuget pack SomePackage.nuspec Path\To\AProject.csproj

 

And it will build a package pulling what info it can from the csproj file (Id, Version, Authors, Files) and supplementing that Metadata from the nuspec file.

Dec 21, 2010 at 6:16 AM

I just tried it and it seems to work.  One problem is that the nuspec has several required fields, so you have to have something in there even though it'll get replaced by the one from the assembly.  Maybe we need a way to read the nuspec with a looser xsd that makes those fields optional.  Also, the nuspec could be omitted altogether if the assembly has everything.

Couple bugs:

  • There is a ReadLine() in there so you have to press enter to run the command
  • It looks like it uses the Summary text for the Language field

Question: does it pick up dependencies from the packages.config file?  I didn't try this.

Dec 21, 2010 at 6:48 AM

Yeah we need a Manifest.ReadWithoutValidation() call essentially :-)

And yeah there is a good chance we could derive all of the essentials from the .csproj file.

 

Whoops thought I had taken the ReadLine() out.  I was testing it from the command line bundling up a random project of mine, and that was my little hack to give me time to attach the debugger so that I could play looser with relative paths and stuff when running it.

Didn't think I was populating language.

 

It currently doesn't do any dependency resolution.

 

But yeah this isn't finished.  Since I moved some of the core around I wanted feedback sooner.  Not even sure if the FileVersionInfo solution is the best (see the stackoverflow question I linked above).

 

Finally:  In the reviewboard discussion we talked about using MSBuild to parse the project file.  How would that port to Mono?  I would like to keep this agnostic enough that it's possible.  If it's not maybe introduce a new "vspack" command or something...?  Or maybe Mono has a better API for reading project files then MSBuild :-)

Coordinator
Dec 21, 2010 at 3:32 PM

Mono has xBuild http://www.mono-project.com/Microsoft.Build which is their port of MsBuild.

Dec 21, 2010 at 6:28 PM

Also now that I'm thinking on it, the original motivation for building a package from a project file was that I knew I could derive a binary from it.  But really we can get most, if not all, of the metadata from the actual .dll.

That might be a better solution:  nuget pack bin\release\mylibrary.dll

Doing it that way would remove the "visual studio" dependency and make it more immediately portable to mono, especially if we used mono.cecil for reading the binary.  That way we wouldn't have to do Assembly.LoadFrom() and leak an app domain :-)

Just a thought.

 

Dec 21, 2010 at 7:45 PM

I can see the pure DLL approach having appeal.  But it also has a downside: you wouldn't be able to figure out package dependencies.  e.g. if I write a little library that uses NHibernate and want to package it, I really want to end up with the proper NH dependency in my package.  Note that technically you don't need the csproj for this, but just the packages.config that NuGet produces in the project folder.

Dec 21, 2010 at 8:02 PM

Yeah I kind of feel that this feature shouldn't be replacing the nuspec file or anything like that, but should serve a very specific case.

What is this packages.config thing?

How do we feel about a "convention over configuration" thing?  Like if a packages.config file is there just use it fulfill the dependencies.

Dec 21, 2010 at 8:09 PM

Packages.config gets created at the root of the project (next to the csproj), and has a list of dependencies, e.g.

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="elmah" version="1.1" />
  <package id="SQLCE" version="4.0.8435.1" />
  <package id="EFCodeFirst" version="0.8" />
  <package id="WebActivator" version="1.0.0.0" />
  <package id="EFCodeFirst.SqlServerCompact" version="0.8" />
</packages>

From that, you can easily feed the dependencies into the PackageBuilder object.

I like the idea of just using it if it's there, but note that it's not in the same folder as the binary (which is typically in bin\debug), so you'd need to look up the parent chain.

Another potentially interesting scenario is including multiple binaries in the package.  I suppose the tool could take multiple DLL's, but only use the first one as the source of metadata, while the rest just go in lib.  Just throwing ideas!

Dec 21, 2010 at 8:22 PM

Ah gotcha; I'm assuming packages.config is something the NuGet infrastructure generates..?

Also before we diverge too much let me lay out the scenario I wanted to enable for our codebases:

Say you have this solution:

MyCoreSolution/

 - MyProjectA/

 - MyProjectB/

You would create a root .nuspec file that each project could use.  It would have the core stuff like License, and Company set.

And then each project could have a post-build step that invokes, say, ..\tools\nuget.exe and points that to bin\release\MyAssemblyName.dll

What we end up with is a package per project in our solution.  We can then build a "meta" package out of that be creating a nuspec with no files, and just a bunch of dependencies which are the individual project generated package files.

So my goal was only to ever say "This won't be a 'composite' package, just pull all the info you need from this one project (or dll) and make me a package".

 

For supporting multiple dll's there is also no reason we couldn't do some kind of .SelectMany() across a bunch of IPackageFileStrategy's.  So you put a <files /> directive in the .nuspec that has some random .dlls or whatevs, and then specify the project / dll on the command line as the "identity" of the package.

 

Does that make sense, or am I just rambling? :-P

Dec 21, 2010 at 9:47 PM

The overall scenario makes sense, though I do think that it needs to support the scenario where MyProjectA happens to be using some existing package, and hence the package for MyProjectA needs a dependency on that package (but maybe the initial implementation can leave that out).

In the end, if we leave aside the implementation details, the basic scenarios we want to enable is to easily build a package from a project, preferably without having to leave VS.  In a sense, this is quite similar to right clicking on a Web App and choosing 'Build Deployment Package'.  We're just building a NuGet package instead of a WebDeploy package.  We could probably add a menu item around the same place for that.

If we can nail this scenario and make it easy to use, I'm sure people will use it.

Dec 21, 2010 at 9:55 PM

Agreed.

 

I'm poking at Mono.Cecil at the moment since the API looks clean and it does what I want (getting to assembly: attributes) without doing bad things (like leaking an app domain).

And the one thing about using .csproj file is that if you rename the assembly in Visual Studio, nuget can still find the resultant dll.  Of course usually when you rename a Class Libraries output you are also renaming the Class Library itself, so maybe it should be one of those "convention over configuration" things again.  Introduce a new packproject command that just looks in the same directory for a .csproj file and assumes that what you meant.  Kind of like what we do today if you don't specify a .nuspec file.

Jan 10, 2011 at 3:30 PM

Still moving ahead with some of these ideas, with the focus being on something we can use internally here and evaluate.

Since I'm doing some things that might be a hard sell, or are more specific to my needs and wants here, I was thinking that it might be nice to have the CommandLine program not only look for ICommand in the current executing assembly, but also in a solution / project specific "plugins" directory.  That way the small core tweaks I've made (IPackageFileStrategy, and IPackageMetadataProvider) can live in the core nuget, but the more esoteric use cases of those (my packassembly command that uses Mono.Cecil to read assembly attributes and info and spits out a package) can be separated out to a "plugin".

From what I can tell the powershell stuff is extensible, but not the NuGet command?  Any thoughts on this?  MEF should make this pretty easy to bang out.

Jan 12, 2011 at 5:54 PM

http://nuget.codeplex.com/SourceControl/network/Forks/ecoffey/PackageFromProject

Moved a lot of CommandLine to CommandLine.Core.  Now CommandLine is just the parser, the program, and the builtin commands.

The program now also recurses up the directory tree looking for a folder named "NuGetPlugins".  If found then that directory is added to the AggregateCatalog.

So now given NuGet.CommandLine.Core, and NuGet.Core you can write a "plugin" that exposes an ICommand.

So now the only thing in NuGet proper is the original refactoring (IPackageFileStrategy and the like), and some small changes to support "plugins"

All of my crazy "create a package from your own dna!" kind of stuff can be contained to "plugins".

 

How do people feel about the way I structured the plugins logic?  Right now it kind of silently loads those plugins, should there be a way to show where this command came from?  Or be more opt-in?  i.e.: "nuget runfrom path\to\plugins\directory commandexposedbyplugin --someoptionforplugincommand"