MSBuild Pack Task

May 3, 2011 at 10:16 PM
Edited May 4, 2011 at 6:27 PM

Hi everyone,

I just spent about two days struggling to have NuGet integrated into my build process.  I had assumed that because NuGet is a packaging and deployment solution that it would be easy, but alas it was not.  The end result is basically a hack that stems from the PackCommand class, which is defined in the NuGet.exe program.

When I first started investigating this I didn't realize that when specifying a project file to NuGet it actually builds the project.  This doesn't make sense to me.  I can build the project myself, so why is NuGet trying to offer a subset of the functionality that MSBuild provides me?  I'd rather just pass it my project file and have it read the info that it needs.  It can determine the output assembly from the project file, or I could even specify that as well, after I build it myself.

As an alternative to passing the project file to NuGet, I tried passing the nuspec file so that NuGet wouldn't attempt to build my project; however, this approach doesn't support replacement tokens or the symbol store, at the very least, so I decided to see what other options were available.

I found an MSBuild task in NuGet's source code, but good luck trying to download it :)  Short of downloading the entire solution myself and building it, I couldn't find it anywhere.  I decided not to try this approach after researching it.  It seems that the result would be much less than optimal anyway, since apparently it has the same shortcomings as passing a nuspec file to the command line tool.

So I decided to create an in-line MSBuild 4.0 task that I could include in my project file to automate the packaging process.  This took over a day to develop given that the way NuGet builds the project causes MSBuild to attempt a complete build of the same project, which fails miserably.  Many different hacks and workounds later, I've got a working solution with the following features:

  1. Single NuGet.targets file that defines the NuGetPack target.  You can import this file into your existing project and call NuGetPack after the Build target (see example below).
  2. Single dependency: NuGet.exe.  It does not use the NuGet MSBuild task and has no other dependencies other than the NuGet command-line tool.
  3. Uses as much of the NuGet packaging code as possible via .NET reflection.
  4. Packaging happens after the build completes, thus it can assume that the assembly has already been built.  It never attempts to build anything, it just reads the data from the project file.  This facilitates a build-once model, which is especially useful if you have auto-incrementing version numbers and you must accomodate multiple packaging and deployment scenarios, not just NuGet.

If I'm way off on this and there's a much simpler solution, then please let me know!

Note that this code is provided AS IS.  I will not be supporting it or fixing bugs.  I'm only including it here so that others may use it as a baseline for automating packaging in their own solutions, and also to give the NuGet team an example of the type of features I'm looking for in an automated packaging solution.

Note that this has not been fully tested yet, although it does appear to work in my scenario.  I'm not making any gaurantees about its quality or stability.

Instructions

To use the NuGet.targets file in your project:

  1. Run the NuGet.exe command-line tool once to create a [Your Project Name].nuspec file.  See step #6 in this blog post for details.  You can leave the tokens if you'd like (although it may not be obvious, $author$ corresponds to AssemblyCompanyAttribute.)
  2. Open your project file and add an Import element at the bottom.  Following that, add a new Target that calls the NuGetPack target after the project builds.  For example:
<Import Project="..\Tools\NuGet.targets" />

<Target Name="CreateNuGetPackage" AfterTargets="Build" DependsOnTargets="NuGetPack">
	<Message Text="Created package: %(NuGetPackages.FullPath)" />
</Target>

Optionally, you can define the following properties in a PropertyGroup before the Import element:

  1. NuGetProgram - Default value expects the NuGet.exe program to be located in the same directory as the NuGet.targets file.
  2. NuGetSpecFile - Default is the full path of the project file with ".nuspec" as the extension instead of the normal project extension.
  3. NuGetProjectFile - Default is the current project file.  (If you add this Import to another imported file instead of directly in the project file, like I often do, then this still works - it will automatically find the project file being built.)
  4. NuGetOutDir - Default is the configuration-dependent output directory of the project.  This is where your packages will be created; e.g., bin\Release\
  5. NuGetBasePath - Default is empty.  An empty value causes the task to use the project directory as the base path of the package.
  6. NuGetTool - Default is False.
  7. NuGetVersion - Default is empty.  An empty value causes the task to pull the version number from the assembly.
  8. NuGetSymbols - Default is True.  This causes the task to build a symbols package containing the .pdb files and source code.  When you push the main package to NuGet later, your symbol package will automatically be pushed to SymbolSource.org.  This will allow users to step into your assembly while debugging if they choose to enable this feature in Visual Studio.  For more information, see the section "What the package Consumer needs to do" in this blog post.
  9. NuGetPackEnabled - Default is True if the NuGetSpecFile exists and the Configuration property is equal to Release.
  10. NuGetExclude - This is actually an item group, not a property.  You can include items to be excluded from the package.

Here's the NuGet.targets file in its entirety:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

	<PropertyGroup>
		<NuGetSpecFile Condition=" '$(NuGetSpecFile)' == '' ">$(MSBuildProjectDirectory)\$(MSBuildProjectName).nuspec</NuGetSpecFile>
		<NuGetProgramName>NuGet.exe</NuGetProgramName>
		<NuGetProgram Condition=" '$(NuGetProgram)' == '' ">$(MSBuildThisFileDirectory)$(NuGetProgramName)</NuGetProgram>
	</PropertyGroup>

	<PropertyGroup>
		<NuGetProjectFile Condition=" '$(NuGetProjectFile)' == '' ">$(MSBuildProjectFullPath)</NuGetProjectFile>
		<NuGetOutDir Condition=" '$(NuGetOutDir)' == '' ">$(OutDir)</NuGetOutDir>
		<NuGetBasePath Condition=" '$(NuGetBasePath)' == '' "></NuGetBasePath>
		<NuGetTool Condition=" '$(NuGetTool)' == '' ">False</NuGetTool>
		<NuGetVersion Condition=" '$(NuGetVersion)' == '' "></NuGetVersion>
		<NuGetSymbols Condition=" '$(NuGetSymbols)' == '' ">True</NuGetSymbols>
	</PropertyGroup>

	<ItemGroup>
		<NuGetExclude Include="**\*.snk" />
		<NuGetExclude Include="**\*CodeAnalysisLog.xml" />
	</ItemGroup>

	<Choose>
		<When Condition=" EXISTS('$(NuGetSpecFile)') AND '$(Configuration)' == 'Release' ">
			<PropertyGroup Condition=" '$(NuGetPackEnabled)' == '' ">
				<NuGetPackEnabled>True</NuGetPackEnabled>
			</PropertyGroup>
		</When>
		<Otherwise>
			<PropertyGroup Condition=" '$(NuGetPackEnabled)' == '' ">
				<NuGetPackEnabled>False</NuGetPackEnabled>
			</PropertyGroup>
		</Otherwise>
	</Choose>

	<UsingTask TaskName="NuGetPack" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
		<ParameterGroup>
			<Packages ParameterType="Microsoft.Build.Framework.ITaskItem[]" Output="True" />

			<ProjectFile Required="True" />

			<OutputDirectory Required="False" />

			<!-- 
			<Properties ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="False" />
			-->
			<Configuration Required="False" />
			<Platform Required="False" />
			<Excludes ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="False" />
			<Tool ParameterType="System.Boolean" Required="False" />
			<Symbols ParameterType="System.Boolean" Required="False" />
			<Version Required="False" />
			<BasePath Required="False" />
		</ParameterGroup>
		<Task>
			<Reference Include="NuGet" />
			<Reference Include="Microsoft.CSharp" />
			<Reference Include="Microsoft.Build" />
			<Reference Include="System.Xml" />
			<Using Namespace="System" />
			<Using Namespace="System.Collections.Generic" />
			<Using Namespace="System.IO" />
			<Using Namespace="System.Linq" />
			<Using Namespace="Microsoft.Build.Evaluation" />
			<Using Namespace="Microsoft.Build.Framework" />
			<Using Namespace="Microsoft.Build.Utilities" />
			<Using Namespace="System.Reflection" />
			<Using Namespace="NuGet" />
			<Using Namespace="NuGet.Commands" />
			<Code Type="Fragment" Language="cs">
				<![CDATA[
const string packageExtension = ".nupkg";
const string symbolsPackageExtension = ".symbols" + packageExtension;
const BindingFlags privateInstanceBinding = BindingFlags.NonPublic | BindingFlags.Instance;

Packages = new TaskItem[Symbols ? 2 : 1];

/* A new project collection is required to workaround an issue with MSBuild's shared project
 * system, which throws when attempting to load the project into the global collection while
 * it's already loaded by the build system in Visual Studio.  (The error occurs on subsequent
 * builds, not the first build.)
 */
using (var projectCollection = new ProjectCollection())
{
	var project = new Project(ProjectFile, null, null, projectCollection);
	
	Type factoryType = Type.GetType("NuGet.Commands.ProjectFactory, NuGet");
	
	object factory = Activator.CreateInstance(factoryType, project);
	
	factoryType.GetProperty("IsTool").SetValue(factory, Tool, null);
	factoryType.GetProperty("Logger").SetValue(factory, new NuGet.Common.Console(), null);
	
	var properties = (IDictionary<string, string>) factoryType.GetProperty("Properties").GetValue(factory, null);
	
	if (!string.IsNullOrEmpty(Configuration))
		properties.Add("Configuration", Configuration);
		
	if (!string.IsNullOrEmpty(Platform))
		properties.Add("Platform", Platform);
	
	/* The following property is required to prevent recursion in the MSBuild target that executes this task, because for 
	 * some unknown reason NuGet insists on building the project even though the output already exists, instead of simply 
	 * reading and packaging the existing output.  (Hopefully by running this MSBuild task in-proc it will at least prevent 
	 * a full rebuild if MSBuild can detect that the output files are still up-to-date.)
	 * 
	 * Update: We can't rebuild the project while a build is currently in progress.  See the mega-comment below for more 
	 * information.
	 */
	//properties.Add("NuGetBuild", "True");
	
	// No longer needed.  See mega-comment below.
	/*
	if (Properties != null)
	{
		foreach (var item in Properties)
		{
			var pair = item.ItemSpec.Split(',');
		
			if (pair.Length == 2)
				properties.Add(pair[0], pair[1]);
		}
	}
	*/
	
	PackageBuilder builder = null;
	
	/* The following function was taken from ProjectFactory.cs and modified to avoid rebuilding the project, which doesn't 
	 * work anyway because MSBuild doesn't allow a project to be built while a build is currently in progress.  Trust me, 
	 * I tried a few things, including calling a separate target from the MSBuild task and even invoking it through MSBuild 
	 * on the command-line via the Exec task, but MSBuild always fails reporting that the project is currently being built.
	 * Regardless, I have no idea why NuGet insists on building the project anyway - it appears to be pointless.  As a "package" 
	 * generator, it should leave the "build" up to build tools that were designed for this, instead of duplicating a subset 
	 * of the build functionality that MSBuild offers (e.g., configuration, platform, other properties, environment, etc.) 
	 * merely to ensure that the output exists?
	 * 
	 * We can build our own projects and then point the NuGet MSBuild task to the project file so that it can determine where 
	 * to look for the output and pull data for the replacement tokens.  It should also have an optional Assembly parameter so 
	 * that the exact output file can be specified, eliminating the need to get the target from the project file, if it differs.
	 */
	Func<PackageBuilder> createBuilder = () =>
		{
			builder = new PackageBuilder();
			
			foreach (var property in properties)
			{
				project.SetProperty(property.Key, property.Value);
			}
			
			project.ReevaluateIfNecessary();
			
			var targetPath = project.GetPropertyValue("TargetPath");
			
			factoryType.GetProperty("TargetPath", privateInstanceBinding).SetValue(factory, targetPath, null);
	
			try
			{
				AssemblyMetadataExtractor.ExtractMetadata(builder, targetPath);
			}
			catch
			{
				factoryType.GetMethod("ExtractMetadataFromProject").Invoke(factory, new[] { builder });
			}
			
			var privateProperties = (IDictionary<string, string>) factoryType.GetField("_properties", privateInstanceBinding).GetValue(factory);
			
			privateProperties.Clear();
			privateProperties.Add("Id", builder.Id);
			privateProperties.Add("Version", builder.Version.ToString());
	
			if (!string.IsNullOrEmpty(builder.Description))
				privateProperties.Add("Description", builder.Description);
	
			string projectAuthor = builder.Authors.FirstOrDefault();
			if (!string.IsNullOrEmpty(projectAuthor))
				privateProperties.Add("Author", projectAuthor);
			
			factoryType.GetMethod("ProcessNuspec", privateInstanceBinding).Invoke(factory, new[] { builder });
			
			if (builder.Authors.Count > 1)
				builder.Authors.Remove(projectAuthor);
	
			builder.Version = VersionUtility.TrimVersion(builder.Version);
			
			// Add output files
			factoryType.GetMethod("AddOutputFiles", privateInstanceBinding).Invoke(factory, new[] { builder });
	
			// Add content files
			factoryType.GetMethod("AddFiles", privateInstanceBinding).Invoke(factory, new object[] { builder, "Content", "content" });
	
			if ((bool) factoryType.GetProperty("IncludeSymbols").GetValue(factory, null))
				factoryType.GetMethod("AddFiles", privateInstanceBinding).Invoke(factory, new object[] { builder, "Compile", "src" });
	
			factoryType.GetMethod("ProcessDependencies", privateInstanceBinding).Invoke(factory, new[] { builder });
	
			if (string.IsNullOrEmpty(builder.Description))
				builder.Description = "Description";
	
			if (!builder.Authors.Any())
				builder.Authors.Add(Environment.UserName);
	
			return builder;
		};
		
	createBuilder();
	
	if (!string.IsNullOrEmpty(Version))
	{
			builder.Version = new System.Version(Version);
	}
	
	Action<int, string> build = (packageIndex, outputFileExtension) =>
		{
			if (string.IsNullOrEmpty(BasePath))
				BasePath = Path.GetDirectoryName(Path.GetFullPath(ProjectFile));
			
			Func<IPackageFile, string> resolvePath = packageFile =>
				{
					var physicalPackageFile = packageFile as PhysicalPackageFile;
			
					if (physicalPackageFile == null)
						return packageFile.Path;
				
					var path = physicalPackageFile.SourcePath;
			
					int index = path.IndexOf(BasePath, StringComparison.OrdinalIgnoreCase);
			
					if (index != -1)
						path = path.Substring(index + BasePath.Length).TrimStart(Path.DirectorySeparatorChar);
	
					return path;
				};
	
			if (!BasePath.EndsWith(@"\"))
				BasePath = BasePath + '\\';
			
			var excludes = Excludes
				.Select(item => new Uri(BasePath)
					.MakeRelativeUri(new Uri(item.ItemSpec, UriKind.RelativeOrAbsolute))
					.ToString()
					.Replace("/", @"\"))
				.Concat(new[] { @"**\*.nuspec", @"**\*" + packageExtension });
			
			PathResolver.FilterPackageFiles(builder.Files, resolvePath, excludes);
			
			string outputFileName = builder.Id + "." + builder.Version.ToString() + outputFileExtension;
			
			string outputPath = Path.Combine(OutputDirectory ?? Directory.GetCurrentDirectory(), outputFileName);
	
			bool isExistingPackage = File.Exists(outputPath);
			
			try
			{
					using (Stream stream = File.Create(outputPath))
					{
						builder.Save(stream);
					}
			}
			catch
			{
					if (!isExistingPackage && File.Exists(outputPath))
					{
						File.Delete(outputPath);
					}
					throw;
			}
			
			Packages[packageIndex] = new TaskItem(outputPath);
	
			Log.LogMessage("Package built successfully: {0}", outputPath);
		};
	
	string projectName = Path.GetFileName(ProjectFile);
	
	Log.LogMessage("Building package for {0}...", projectName);
	
	build(0, packageExtension);
	
	if (Symbols)
	{
		Log.LogMessage("Building symbols package for {0}...", projectName);
	
		factoryType.GetProperty("IncludeSymbols").SetValue(factory, true, null);
		
		createBuilder();
		
		build(1, symbolsPackageExtension);
	}
}
]]>
			</Code>
		</Task>
	</UsingTask>

	<Target Name="NuGetPack" Condition=" $(NuGetPackEnabled) == True ">

		<!-- 
			Workaround for an in-line task bug that causes Reference assemblies to be loaded from the 
			MSBuild bin path even though the full assembly path is specified in the Reference element
			and the assembly is successfully loaded during pre-processing of the UsingTask element (I know 
			this because the binding error shows the full assembly display name, not the specified path.).
		-->
		<Copy Condition=" '$(BuildingInsideVisualStudio)' == False AND !EXISTS('$(MSBuildBinPath)$(NuGetProgramName)') "
			SourceFiles="$(NuGetProgram)" DestinationFolder="$(MSBuildBinPath)" />
		<Copy Condition=" '$(BuildingInsideVisualStudio)' == True AND !EXISTS('$(DevEnvDir)\$(NuGetProgramName)') " 
			SourceFiles="$(NuGetProgram)" DestinationFolder="$(DevEnvDir)" />

		<PropertyGroup>
			<_NuGetExclude>@(NuGetExclude->'%(FullPath)')</_NuGetExclude>
		</PropertyGroup>

		<NuGetPack ProjectFile="$(NuGetProjectFile)"
							 Configuration="$(Configuration)" Platform="$(Platform)"
							 OutputDirectory="$(NuGetOutDir)" Excludes="$(_NuGetExclude)"
							 BasePath="$(NuGetBasePath)" Version="$(NuGetVersion)"
							 Tool="$(NuGetTool)" Symbols="$(NuGetSymbols)">
			<Output TaskParameter="Packages" ItemName="NuGetPackages" />
		</NuGetPack>

	</Target>

</Project>

- Dave

May 4, 2011 at 12:32 AM

I just updated the source code in the first post to fix a bug with the Excludes functionality.

May 4, 2011 at 12:59 AM

Hi Dave, your code looks impressive, but its too late for me to dive into it in detail. I build my packages using a simple post-build script specified in the properties of the projects resulting in nuget packages.

I have a package per project, the project contains the nuspec file and has as base name the name of the project.

$(SolutionDir)Tools\NuGet.exe Pack  $(ProjectName).nuspec -Verbose -BasePath $(ProjectDir) -OutputDirectory $(SolutionDir)Dist\NuGet-Factory"

May 4, 2011 at 1:20 AM

Hi,

Yes, that was one option that I had considered but I decided against it.

  1. Does your .nuspec support replacement tokens from assembly attributes?
  2. Are dependencies discovered automatically from the packages.config file?
  3. Are you automatically building a symbols package for SymbolSource.org?

- Dave

May 4, 2011 at 1:23 AM

Hi Dave:

1:  No, 2: No, 3: No :-)

I will have a good look at your approach when I'm awake again:-)

Seems like you did a good job! Although sometimes simplicity can be good as well...

May 4, 2011 at 1:24 AM
Edited May 4, 2011 at 1:25 AM

I'm all for simplicity if it saves me two days of work ;)

Edit: Without sacrificing features.

May 4, 2011 at 7:20 AM

I've just updated the source code in the first post to fix a bug with resolving the NuGet reference assembly when building inside Visual Studio.

May 4, 2011 at 5:37 PM

I've just updated the source code in the first post to fix a bug caused by the NuGetBuild property usage.  Since it's no longer required I've removed it.

May 4, 2011 at 5:51 PM
Edited May 4, 2011 at 5:52 PM

Just fixed another bug that causes Visual Studio builds to fail when the copied NuGet.exe assembly is already loaded.
A consequence of this fix is that the version of the copied NuGet.exe assembly will always remain the same unless you update it manually.

Developer
May 4, 2011 at 6:10 PM

Have you looked at integrating these changes with the NuGet.MsBuild project that is part of the source tree. It would make distributing your code a whole lot easier.

May 4, 2011 at 6:30 PM

Just fixed another bug that occurs on subsequent builds within Visual Studio.
It was caused by NuGet loading the MSBuild project into the global project collection, which caused a conflict with the project that Visual Studio had already loaded into the same collection.

May 4, 2011 at 6:33 PM

Do you mean a patch?  I'll certainly consider doing that.  Would you consider making the NuGet.MSBuild library available for download on the CodePlex page?  Or at least, via NuGet.

- Dave

Developer
May 4, 2011 at 7:29 PM

A typical contribution involves creating a fork and sending a code review for the changes. Here's a brief guide on our contribution steps: http://nuget.codeplex.com/documentation?title=Contributing%20to%20NuPack, More details are available in the documentation section.

And I don't see any issues distributing it via the feed.

May 4, 2011 at 7:55 PM

Great thanks, I'll take a look.