Simplifying the nuspec and package format

Sep 3, 2010 at 10:07 PM

Currently, our nuspec and package format are on the complex side:

  • The nuspec has separate sections to specify the various files: resource, content, config transform, assemblies
  • Those each turn into a different type of 'relationship' in the OPC package

We're thinking of moving to a much more convention over configuration approach, where it's just a bunch of files, and their semantic varies based on what folder they're in.  This would mean that neither the .nuspec nor the .nupack would know anything about the various types of files.  It's only when we install it and apply it to specific project that we would understand the special semantic of the folders.

The best analogy it trying to zip up an ASP.NET site.  The zip has no special information about things being web files vs code files vs binary files.  It's only the ASP.NET runtime that interprets App_Code, bin, etc... as special.

The main reasoning is that this will lead to a generally simpler model with less concepts.  Note that this would not affect how NuPack is used in VS in any way.

Any thoughts on this?

Sep 3, 2010 at 10:15 PM

The simplier the better.  I have been working with creating a package and the nuspec and my stomic turns when I look at the internals of the opc file.  I think some conventions would be great. I am hoping this would mean you can potentially reduce the nuspec file to just the metadata about the package instead of having to manage the file locations and target dirs.

Editor
Sep 3, 2010 at 10:17 PM
So this is with respect to where things ARE not where things end up Going, right?

--
Scott Hanselman - Tiny keyboard, tiny email - Sent from some kind of phone

On Sep 3, 2010, at 2:09 PM, davidebb <notifications@codeplex.com> wrote:

From: davidebb

Currently, our nuspec and package format are on the complex side:

  • The nuspec has separate sections to specify the various files: resource, content, config transform, assemblies
  • Those each turn into a different type of 'relationship' in the OPC package

We're thinking of moving to a much more convention over configuration approach, where it's just a bunch of files, and their semantic varies based on what folder they're in. This would mean that neither the .nuspec nor the .nupack would know anything about the various types of files. It's only when we install it and apply it to specific project that we would understand the special semantic of the folders.

The best analogy it trying to zip up an ASP.NET site. The zip has no special information about things being web files vs code files vs binary files. It's only the ASP.NET runtime that interprets App_Code, bin, etc... as special.

The main reasoning is that this will lead to a generally simpler model with less concepts. Note that this would not affect how NuPack is used in VS in any way.

Any thoughts on this?

Sep 3, 2010 at 10:20 PM
+1 for simplicity. :)
-d

On Fri, Sep 3, 2010 at 4:07 PM, davidebb <notifications@codeplex.com> wrote:

From: davidebb

Currently, our nuspec and package format are on the complex side:

  • The nuspec has separate sections to specify the various files: resource, content, config transform, assemblies
  • Those each turn into a different type of 'relationship' in the OPC package

We're thinking of moving to a much more convention over configuration approach, where it's just a bunch of files, and their semantic varies based on what folder they're in.  This would mean that neither the .nuspec nor the .nupack would know anything about the various types of files.  It's only when we install it and apply it to specific project that we would understand the special semantic of the folders.

The best analogy it trying to zip up an ASP.NET site.  The zip has no special information about things being web files vs code files vs binary files.  It's only the ASP.NET runtime that interprets App_Code, bin, etc... as special.

The main reasoning is that this will lead to a generally simpler model with less concepts.  Note that this would not affect how NuPack is used in VS in any way.

Any thoughts on this?

Read the full discussion online.

To add a post to this discussion, reply to this email (nupack@discussions.codeplex.com)

To start a new discussion for this project, email nupack@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Sep 3, 2010 at 10:27 PM

Note that we would still be using OPC, which has benefits in term of package signing and standard metadata.  However, we would stop relying on OPC relationships to track our files, and instead do it ourselves.  What we're doing with OPC relationships today is a bit artificial and provides little tangible value.

I think we would still keep the file location concept in the nuspec because it is useful for situation where you want together specific files scattered in various places (we got that feedback early on).  However, I could envision having a default such that if you omit that section entirely, we include all the files that are under the same folder as the .nuspec file (or something along those lines).

Scott, I don't quite understand your question.  We need to be very precise when we talk about file locations, as it can mean 4 different things:

  1. Location of the files on your disk before you build the package
  2. Location of the files inside the package
  3. Location of the files inside the solution level 'packages' folder
  4. Location of the files inside a specific project

But generally, this proposed change is about making all those things much more similar (to each other) than they are now.

Sep 5, 2010 at 8:11 PM

From a community / open source acceptance perspective and frankly from a "keeping-the-developers-sane" one, simplicity is best. If there's information we already know, don't ask the nuspec creator for it (like a date/time when the package was created). I'd like to see the proposal for the nuspec change and compare it to what's already there and what a user would need to get the job done.

Developer
Sep 5, 2010 at 8:30 PM

Latest changes

http://nupack.codeplex.com/wikipage?title=Nuspec%20Format

Sep 5, 2010 at 11:19 PM

The spec is fine, to me the big value is putting xome conventions into the nupack.exe

Developer
Sep 5, 2010 at 11:22 PM

Conventions don't exist at the at the package building level, files are just files until they are actually being installed into the project. At that point we take lib\* to be assembly references tools\* is added to the path and content\* is copied into the project. This is all mentioned in the spec under the conventions section.

Sep 5, 2010 at 11:42 PM

I read the spec, I am better at putting together a poc and describing a specific scenerio. I am thinking about a nuspec file that would pick up tools and libs by specifing the parent folder.

Developer
Sep 5, 2010 at 11:53 PM

So something like this:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
    <metadata>
        <version>1.0</version>
        <id>MyPackage</id>
        <language>en-US</language>
        <description>Eric's awesome package</description>
        <authors>
            <author>Eric Hexter</author>
        </authors>
    </metadata>
    <files src="..\parent\**\*.*" />
</package>

With a folder structure like:
 parent/
    lib/
    tools/ 
    manifest.nuspec

Is that what you have in mind?

Sep 6, 2010 at 7:50 AM

If the .nuspec is a sibling to the lib/tools folders, then it should just be src="**\*.*", right?

And also, we discussed that in the absence of a <files> tag, we default to **\*.*, so in your example, it could just be omitted.

Sep 6, 2010 at 6:44 PM

That is what I was thinking, minimize the amount of spec churn by following the conventions.

Sep 7, 2010 at 3:23 PM

The only addition I would see adding is a way for the nupack.exe to take the spec file and the version number.  From the aspect of automating the publishing this would allow creating the spec once and not having to change it as part of the build process.  The version number could override the value in the template nuspec file. 

Sep 7, 2010 at 3:32 PM
in the ruby world we do this by just reading in a text file

typically
'VERSION'

http://github.com/rails/rails/blob/master/rails.gemspec#L1

I have seen some shops that even set up shell scripts to increment the thing.

one nice side effect of having the spec defined in an executable language. :)

-d

On Tue, Sep 7, 2010 at 9:23 AM, erichexter <notifications@codeplex.com> wrote:

From: erichexter

The only addition I would see adding is a way for the nupack.exe to take the spec file and the version number.  From the aspect of automating the publishing this would allow creating the spec once and not having to change it as part of the build process.  The version number could override the value in the template nuspec file. 

Read the full discussion online.

To add a post to this discussion, reply to this email (nupack@discussions.codeplex.com)

To start a new discussion for this project, email nupack@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Sep 7, 2010 at 3:53 PM
It just seems like writing that version number to disk, when it is already in memory in a build script as well as written into our assemblies seems like just one more step that can be skipped.   
 
You hit an interesting point.. what if the specs were written in powershell instead of XML.... that could be interesting. 
Sep 7, 2010 at 4:02 PM

@erichexter: We had an email discussion on specs and Dru has proposed using PowerShell for the specification. I thought it was sort of smart because it's an executable spec instead of configuration but it was shot down pretty quickly by MSFT in that we would not make the any part of the system dependent on PS but rather use whatever was in the framework.

Coordinator
Sep 7, 2010 at 4:46 PM

The reason we don't want to rely on PS for core system functionality is that we want this stuff to be runnable as part of a website in Medium trust.

For example: You go to CodePlex.com and upload your latest download zip. While entering the details, you check a box that says (Make this a NuPack package). That allows you to fill out more metadata in the form of the .nuspec fields. Then CodePlex.com can automatically create a package for you.

If the spec format was PS, this simple scenario becomes very difficult. CodePlex.com would have to host PS in order to execute the spec.

Sep 7, 2010 at 7:12 PM

I just walked through package creation with the latest code and that is a nice simple experience, having the files automatically picked up from the nuspec directory. 

Developer
Sep 7, 2010 at 7:46 PM

You can even get intellisense for the nuspec if you point your xml file to the nuspec xsd schema :).

Sep 7, 2010 at 8:26 PM

One concern I have is that if the assembly files are copied to ‘lib’ and become project references, those files can be downloaded directly through the browsers. Do we need to worry about that? On one hand, these assemblies are already publicly available, so we’re not revealing any private code. However, there may be cases where web app authors do not want to reveal what assemblies/packages are being used.

So, should we drop a web.config file inside ‘lib’, by default, to block access to the assemblies?

Sep 7, 2010 at 8:29 PM

We should have the installer copy the xsd schema file to the private folder under VS.NET install folder so that it shows up automatically in the property window.

Sep 7, 2010 at 8:30 PM

@dotnetjunky: Not sure I follow? You add a package to your project and it references an assembly. That assembly has to be deployed with the app but it won't be in the lib folder on deployment, it'll be wherever the other assemblies live (bin dir, GAC, whatever). NuPack and deployment are orthagonal concepts so there's no need to block the lib dir. Or am I not following something here?

Sep 7, 2010 at 8:39 PM

Oh yeah, sorry I was confused. The assemblies are not copied to the lib folder of the projects. They are only installed at the solution level.

However, I just realized that the solution .sln file can be located inside a project folder. This happens when the “Create directory for solution” checkbox is unchecked. This is really an edge case though. So we may not need to worry about that.

Coordinator
Sep 7, 2010 at 9:49 PM

I'm not so sure we can dismiss this as an edge case. It's certainly not the structure for a solution we would recommend. But it's easy for people to accidentally get in this boat if they don't check "Create Solution Folder..." when creating a new project.

We should make sure to investigate what happens in this scenario and whether we should mitigate it. For example, if we thought it was important, we could detect the case where a lib folder that we created is within a web project folder and add a web.config in there that blocks access to the dll files. I'm not saying we should do this, just that it would be a potential mitigation.

Sep 7, 2010 at 9:53 PM
I am thinking this might be a better v2 feature to go after.
-d

On Tue, Sep 7, 2010 at 3:49 PM, haacked <notifications@codeplex.com> wrote:

From: haacked

I'm not so sure we can dismiss this as an edge case. It's certainly not the structure for a solution we would recommend. But it's easy for people to accidentally get in this boat if they don't check "Create Solution Folder..." when creating a new project.

We should make sure to investigate what happens in this scenario and whether we should mitigate it. For example, if we thought it was important, we could detect the case where a lib folder that we created is within a web project folder and add a web.config in there that blocks access to the dll files. I'm not saying we should do this, just that it would be a potential mitigation.

Read the full discussion online.

To add a post to this discussion, reply to this email (nupack@discussions.codeplex.com)

To start a new discussion for this project, email nupack@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Coordinator
Sep 7, 2010 at 9:56 PM

Most likely, I'd agree unless we uncover some gaping security hole. For the most part, if all we're potentially exposing are 3rd party OSS assmeblies, so what. My concern would be the organization using packages internally to share proprietary code amongst themselves. Hopefully though, they'd have the smarts not to get into this situation in the first place. :)

Sep 7, 2010 at 10:00 PM

Agreed that we should not ignore this.  Are you sure that IIS even allows .dll downloads by default?

Coordinator
Sep 7, 2010 at 10:09 PM

It appears to. I just checked by putting a dll outside of the bin folder in haacked.com and I downloaded it. (http://haacked.com/foo.dll)

Sep 7, 2010 at 10:12 PM

Ha ha, sucker!  Now I have your secret file too! :)

Sep 8, 2010 at 4:09 AM

I'm lost in the thread. What difference does it matter if a user can "hack" a url to download a dll? We're talking about promoting open source. That's like taking Linux and putting up on a torrent and saying "aw3s0m3 l33t 0p3r8ting syst3m w/src c0dez". Why is this an issue?

P.S. I just haacked haacks sekrit filez too. Say that 3 times fast!

Coordinator
Sep 8, 2010 at 5:07 AM
Edited Sep 19, 2010 at 12:56 AM

I gave a scenario where this is bad. Suppose you are Secret Corp and you are using packages internally from a secured internal feed. And your company is sharing private packages to build an app. You wouldn’t want those assemblies to be exposed publicly.

After all, just because you’re making use of OSS doesn’t mean you want all the source code you write to be accessible publicly. Not only that, you wouldn’t want people to find out which version of an OSS assembly you’re using in case it has security flaws.

Developer
Sep 8, 2010 at 5:52 AM

This thread seems to have diverged from the main topic. I'd like to bring it back by asking for any feedback on the nuspec schema itself. Are we missing any elements? Is the nuspec easy to create? Are we missing any scenarios? Eric brought up being able to change the version per build. We should try to find more of these scenarios and drive towards a solution.

David

Sep 8, 2010 at 11:58 AM

I think this new thread should spin off to package delivery.

I was interested in a single command line option to override the version in a nuspec during package creation.