Filter Packages for Client / Project

Feb 8, 2012 at 6:03 PM

As the number of packages grows, and the number of clients and project types also grows, NuGet is going to need to have the ability to filter packages based on the client and contextual project.

After much in-person discussion, we're arriving at the requirement for NuGet clients to specify additional information via HTTP headers when requesting the list of packages available in the feed.  The data that has been identified as important to send includes:

  1. IDE/Client identifier and version.  For instance:
    1. VS/10.0 (NuGet Package Manager Console 1.6.21215.9133)
    2. VS/10.0 (NuGet Add Package Dialog/1.6.21215.9133)
    3. WebMatrix/2.0
    4. NuGet Package Explorer/2.5.0.0
  2. NuGet.Core version used by the client.  For instance "NuGet/1.6.21215.9133"
  3. .NET Framework/Version. For instance ".NET Framework/4.5"
  4. Project Type / Version. For instance "AspNetWebPages/2.0"
  5. Special client capabilities.  For instance "PowerShell/2.0"

If all represented in a User-Agent type string, an example could be: NuGet/1.6.3 (.NET 4.5; AspNetMvc 4.0; VS 10.0; PowerShell 2.0)

We need to determine which pieces of this would have filtering applied in which ways and whether we'd send all of this in one string, or as separate headers.

 

Feb 8, 2012 at 6:30 PM

It needs to be possible to send multiple project types from the client if there hasn't been an explicit project type choice made.  For instance, if I'm editing a Web Site that contains several flavors of HTML-based applications, it could be possible for the client to send:

  • AspNetWebPages
  • AspNetWebForms
  • Asp
  • HTML
Feb 9, 2012 at 9:37 AM

After further discussions in JabbR, @anglicangeek came to the conclusion that there's actually a really primitive "capability" that is underneath the multiple flavors of web projects: Compilation.

A client such as WebMatrix cannot perform compilation, and there is no project system either.  This is also true when editing a Website in Visual Studio.  On the other hand, when editing an ASP.NET MVC application or any other "project," there is a project system, and compilation can occur.

With every project type conversation we've had, it keeps getting more and more complicated, and we need to keep this very simple.  We don't want to get into the mess of hierarchical project types and managing that big nasty system.  Instead, it's looking more like we would do something like the following:

  • User-Agent: {0}/{1} ({2})
    • {0} = Client Name and Version, such as "WebMatrix 2.0 Beta" or "VS 10.0 SP1"
    • {1} = NuGet.Core version
    • {2} = OS Name/Version
    • This is possible today using NuGet.HttpUtility.SetUserAgent and NuGet.HttpUtility.CreateUserAgentString
  • QueryString: includes the "targetFramework" parameter
    • This is possible today using ISearchableRepository.Search
  • NuGet-Target: Website|Project
    • WebMatrix would send "Website"
    • VS would send "Website" when targeting a Website, or "Project" otherwise
  • NuGet-Capabilities:
    • Comma-Separated list of capabilities the client can perform
    • Only known value at this time is "PowerShell"
    • When the client has no capabilities to claim, "None" can be specified as a "null" value, since the absence of this header indicates that no capability filtering/relevance should be performed.

With this approach, we would not need granular project types such as AspNetWebPages or AspNetMvc.  Instead, we just specify "Website" or "Project" as the NuGet-Target.

Example requests:

WebMatrix

  • URL: http://nuget.org/api/v2/Search()?$top=5&searchTerm='jQuery'&targetFramework='net40'&includePrerelease=false HTTP/1.1
  • User-Agent: WebMatrix 2.0/1.6.21205.9031 (Microsoft Windows NT 6.1.7600.0)
  • NuGet-Target: Website
  • NuGet-Capabilities: None

VS 10.0 SP1 Editing an ASP.NET MVC Application

  • URL: http://nuget.org/api/v2/Search()?$top=5&searchTerm='jQuery'&targetFramework='net40'&includePrerelease=false HTTP/1.1
  • User-Agent: VS 10.0 SP1 NuGet Add Package Dialog/1.6.21205.9031 (Microsoft Windows NT 6.1.7600.0)
  • NuGet-Target: Project
  • NuGet-Capabilities: PowerShell
Feb 9, 2012 at 6:28 PM

I don't think Compilation is the one big primitive capability. In fact, in most cases it's not that relevant in NuGet:

  • Binary references and static files work equally well in WAPs and Sites
  • When dealing with a Site, NuGet copies source files under App_Code instead of their original location. So they end up working either way

Obviously, there are important differences between WAPs and Sites, but we need to define more clearly what that implies in term of what packages can/cannot be used in each case.

So the specific question is: what are you planning to filter based on the client specifying 'NuGet-Target: Website|Project'?

Feb 9, 2012 at 6:36 PM

I agree with David that Comnpilation doesn't seem relevant in NuGet. NuGet doesn't compile projects; it only copy content files and assembly references.

One type of filter that can apply to Website vs. Project is that many packages rely on install.ps1 to modify the project files (e.g. adding custom tasks, etc.)

Feb 9, 2012 at 6:39 PM

Right, it's not compilation, it's the type of "project" (whether an actual proejct or site). Source code files should be fine if NuGet handles them well. What do we plan to filter? The idea is that if a package would work in a website in WebMatrix or VS, it would be included.

I think the big question for me at this point is, does WebMatrix have different rules for what a site can have than VS?

Feb 9, 2012 at 9:14 PM

C# files in the App_Code dir are compiled by the server, they would work on WebMatrix. Packages stored in App_data\Packages and DLL’s in bin will also work in WebMatrix. That's as far as it gets.

Feb 9, 2012 at 9:18 PM

Walter, are there any differences between building a website in WebMatrix versus VS? Any different file type handling? Differences in design-time support? That sort of thing? As I recall, we've always said you should be able to move your websites between WMx and VS, so my guess is no, but I'm no WMx expert.

Feb 9, 2012 at 9:54 PM

All static content file types are OK with WebMatrix: HTML, CSS (LESS, SaSS, ScSS, ), Jscript, CSHTML, CoffeScript, SQL Script, Web.config, Jade, TXT, XML. You can definitely move a site from WMx to VS as long as it is not a PHP or Node site. But the opposite does not necessarily apply. WebMatrix does not provide a build system for compiling code files. It uses a Web site model as opposed to a web application model.

Examples that won’t work:

 -          Any app that follows an source code file structure that requires compilation such as MVC web apps directory structures do not work. They have source code in directories other than App_Code.

-          ASP.NET master pages or user controls which use source files for their implementation and would require compilation would not work.

-          Any site apps that have source code in subprojects, or any directory other than App_Code (what IIS can compile), won’t work in WebMatrix.

-          PowerShell will not work in WebMatrix.

In terms of Design-time support, WebMatrix’s Editor will open the CS files (or VB files) in App_Code and all other file types mentioned above in the site dir structure for editing without problems.

Feb 9, 2012 at 10:00 PM

Walter, I think you misunderstood Drew's question. He's asking about WebMatrix sites vs Web Sites in VS. Obviously, WAPs are very different beasts.

And it should be the same, because the runtime is identical. VS/WebMatrix are just acting as fancy editors in those scenarios (unlike with WAPs). Obviously, the design time support is quite different, but I'm not sure that should affect NuGet.

This doesn't mean that any package that works in a VS WebSite will work in WebMatrix, because of the PowerShell angle. I'm not sure is there is anything else besides PowerShell.

Feb 9, 2012 at 10:02 PM

So maybe we just ditch the whole project type specification.  We'd still be able to filter on Target Framework and PowerShell.

WebMatrix

  • URL: http://nuget.org/api/v2/Search()?$top=5&searchTerm='jQuery'&targetFramework='net40'&includePrerelease=false HTTP/1.1
  • User-Agent: WebMatrix 2.0/1.6.21205.9031 (Microsoft Windows NT 6.1.7600.0)
  • NuGet-Capabilities: None

VS 10.0 SP1 Editing an ASP.NET MVC Application

  • URL: http://nuget.org/api/v2/Search()?$top=5&searchTerm='jQuery'&targetFramework='net40'&includePrerelease=false HTTP/1.1
  • User-Agent: VS 10.0 SP1 NuGet Add Package Dialog/1.6.21205.9031 (Microsoft Windows NT 6.1.7600.0)
  • NuGet-Capabilities: PowerShell
Feb 9, 2012 at 10:08 PM

Any nuget package that uses VS extensibility or other VS features will not work in WebMatrix. Not sure if such nugets exist other than those that use Powershell.

Feb 9, 2012 at 10:43 PM

Ideally WebMatrix can use a default filter that removes WAP-like packages (example: MvcScaffolding). But, yes, WMx will also have the ability of querying for "all", which in its context means (everything but powershell), that query may be the one you have above <?>. To re-afirm the comments above regarding VS Sites and WMx Sites: they are compatible.


Feb 9, 2012 at 10:55 PM

The reason WMx will have the "All" query is because users could potentially get a package that requires compilation, proceed to compile in the command line or VS and come back to WMx to publish. This would be a very small percentage of the WMx users, hence the need for a more refine default filter.

Feb 10, 2012 at 12:56 AM

It really seems like we're determining that project-type filters are just not feasible.  That means we would need to filter the packages out for different reasons.  Let's take the MvcScaffolding example; here are its contents:

\content\InstallationDummyFile.txt
\tools\*
\tools\init.ps1
\tools\install.ps1
\tools\registerWithMvcTooling.ps1

Because of this package's PowerShell requirement, we should be able to filter the package out because the NuGet-Capabilities header would not specify PowerShell.  There's nothing else in this package that could cause us to reject it for WebMatrix.

Another package that would logically get filtered out is MvcContrib, and the Mvc3Futures package that it depends on.  Both of these packages merely contain DLL files in their lib folders.  These would not match any filter heuristic and would therefore show up in WebMatrix.  I'm not sure what we could do to prevent that.

Feb 10, 2012 at 1:34 AM

Yes, I agree with Jeff.

There is a distinction between packages that just can't work in WebMatrix (MvcScaffolding) and packages that technically would work, but are not so likely to be used (like MvcContrib).

Please do note that it is possible to run Mvc apps in WebSites. You'd end up with controllers in App_Code, and you might need a custom controller factory. It's just not something many people do. But if you did, MvcContrib would probably work.

To me, the biggest value of this new filtering logic is the target framework part. Being able to write a Windows Phone app in VS and only see packages that apply to that would be a vast improvement over today's story (where we end up showing the packages, but failing at install time).

When it comes to WebMatrix, I think the best approach is to always look at specific packages and try to make sense of them, as Jeff does above (e.g. would they work, and if not can we detect it). Trying to solve the problem by looking for general rules without looking at specific packages is I think much harder. If we reason through enough specific packages, we should end up with the right general rules.

Feb 10, 2012 at 1:42 AM

The question is then just: Does WebMatrix need to send us anything on the wire for us to be able to filter packages out, even if it's extra metadata we add to packages by hand?  Or is this just a new curated feed (like what MyGet offers), with no automatic intelligence at all?  Or neither?

Feb 10, 2012 at 1:47 AM

I think we need more specific package data. If this came up as an issue, I assume that the consensus was that the unfiltered feed was showing too many packages. If that's the case, then can we get a list of the top 20 packages that show up but that are deemed inappropriate for WebMatrix, and for what specific reason (e.g. broken vs unlikely use).

I really think that this would be useful data to help reason through potential automated filtering, or whether a hand curated feed is the only viable approach to get a clean WebMatrix feed.

Feb 10, 2012 at 1:50 AM

Yeah, that makes sense.  I asked @waltero earlier today if he could pick some packages that should not show up in WebMatrix and tell us what within the package contents/metadata we could use to make the decision.  With that data, we can determine if the decision points are solid enough to be able to assign a "required capability" to those packages that WebMatrix would lack.

Feb 10, 2012 at 7:00 PM

I went through close to 40 nugets, the top 20 plus others. PowerShell continues to be the number one item to avoid. However, here are some items I found would require clarification:

  1. Some packages such as JQuery has a tools dir with install.ps1 and uninstall.ps1. We would not want to have JQuery filter out because of the ps scripts. Would that be the case? This question applies to other packages with similar content.
  2. Some packages install T4 Templates. My guess is that in general these would not work in WMx. It is more of a VS feature, WMx does not have the ability to run the tool. Could these be filtered out? Or there is a manual way a WMx user could make them do their magic easily outside WMx (other than using VS to run the tool)?

Could you clarify?

Thanks!

Walter.

Feb 10, 2012 at 7:12 PM
I'd like to make an interesting observation. Limiting out packages based on package abilities could have an intended/unintended effect of packages looking like this:
  • elmah.core
  • elmah.content (depends on core)
  • elmah.powershell (depends on core)
  • elmah (depends on content and powershell)
I wasn't under the impression that we wanted to go this way with nuget though.

But it begs the question, what if all four of those were really the same package with the user deciding what they wanted run on install?
  • Install-Package elmah -core
  • Install-Package elmah -content
  • Install-Package elmah -powershell
  • Install-Package elmah

Feb 10, 2012 at 10:08 PM
waltero wrote:

I went through close to 40 nugets, the top 20 plus others. PowerShell continues to be the number one item to avoid. However, here are some items I found would require clarification:

  1. Some packages such as JQuery has a tools dir with install.ps1 and uninstall.ps1. We would not want to have JQuery filter out because of the ps scripts. Would that be the case? This question applies to other packages with similar content.
  2. Some packages install T4 Templates. My guess is that in general these would not work in WMx. It is more of a VS feature, WMx does not have the ability to run the tool. Could these be filtered out? Or there is a manual way a WMx user could make them do their magic easily outside WMx (other than using VS to run the tool)?

Could you clarify?

Thanks!

Walter.

For 1, yes, jQuery would work, as it used PowerShell, but doesn't require it (and works without it).

For 2, I assume they wouldn't work, as there is no T4 host in WMx. @davidebbo know more about this than me, so he can correct me if I'm wrong. As far as capability goes, I suppose we could add it this as one (in addition to PowerShell), if we feel it's important enough. It's not quite as bad as PowerShell, since it will install, but it obviously won't work as intended. Is this something we'd need to filter out immediately, or wait to see if it causes problems and confusion with folks?

Feb 10, 2012 at 10:29 PM
ferventcoder wrote:
But it begs the question, what if all four of those were really the same package with the user deciding what they wanted run on install?
  • Install-Package elmah -core
  • Install-Package elmah -content
  • Install-Package elmah -powershell
  • Install-Package elmah

The goal is to avoid needing to create different packages if it's really about required capabilities. For the ELMAH case (and ignoring the storage-specific packages), I'd expect an ELMAH.Core with just libs, and an ELMAH that might *use* PowerShell, but not *require* PowersShell, so that it worked in VS, WMx (and just about everywhere). 

We definitely don't want to punish the consumer, even if it means a little extra work spec'ing the package *only when the default capability detection wouldn't work*. The NuGet client will know it's capabilities, and will filter or order packages accordingly. Package authors will have to be careful about limiting the reach of their packages based on this (the limit already exists, since the non-VS clients already exist, but I don't think many folks think about it.)

Long and short: install-package ELMAH needs to just work; I should never need to know more than that as a consumer. If for some reason ELMAH really cannot work for my current NuGet client, than it would be great to filter it out.

Feb 10, 2012 at 10:42 PM
ELMAH was provided as an example as part of the thought exercise. It really could apply to any package. moving on...

My $0.02 - I'm fine with all of this filtering by default as long as I have a switch to remove the filter (in the console / command line / other places). Don't constrain me without giving me a way to remove the constraint. It sounds like you will allow for that from some of the email traffic, but I just wanted to make sure it was out there.

Feb 10, 2012 at 10:48 PM

@ferventcoder, Yeah, I know ELMAH is not a great example. I guess what I'm saying is that there packages that are broken today, but presented as options anyway because we don't have a way for clients to indicate what they're capable of (and in one case we do, with target FX, and we're not making good use of it). The idea is to fix that, so that stuff that really is broken either doesn't show up by default, or is given less priority in how it is displayed. In short: stuff that works now all still works, stuff that's broken is harder to install.

We will have an escape hatch of some sort for sure. That escape hatch will likely vary by client. In VS, you always have the console, for sure. And we still need to decide whether this is a filtering mechanism (it doesn't show up) or a relevance/ordering mechanism (as the bottom of the list, maybe with some visual indicator it's not compatible).

Whatever we do shouldn't make life harder for consumers, even advanced consumers.

Feb 10, 2012 at 10:49 PM

I will feel better if we add the T4 capability to filter them. It is hard to see how many there are, it looks like few, but some are popular and quite involved for adding to WebMatrix. Besides, WebMatrix downloads of these will skew their donwload stats :) Thanks!

Feb 10, 2012 at 10:54 PM

T4 templates indeed wouldn't work in WebMatrix. But note that a package like T4MVC wouldn't work in a WebSite in VS either, as it relies on lots of WAP only concepts. But that doesn't mean that it's true of all T4 packages. So that's clearly a case where just looking at the package file list won't answer all the filtering questions. T4MVC might need to explicitly declare that it requires WAP if we don't want it to show up in VS WebSites.

 

@ferventcoder: I agree that there needs to be a way to turn off filtering, as it will inevitably over filter in some scenario.

Feb 10, 2012 at 10:59 PM
davidebbo wrote:

T4 templates indeed wouldn't work in WebMatrix. But note that a package like T4MVC wouldn't work in a WebSite in VS either, as it relies on lots of WAP only concepts. But that doesn't mean that it's true of all T4 packages. So that's clearly a case where just looking at the package file list won't answer all the filtering questions. T4MVC might need to explicitly declare that it requires WAP if we don't want it to show up in VS WebSites.

@davidebbo, I think we can defer the WAP vs website issue for now. We have that problem today in VS, and it wouldn't change if we added T4 as a capability, since VS would continue to support it. But at least anything with T4 would be excluded in WMx.

(I do think we might get to the point where the filtering needs to have some more information about the project system, but I also don't think that's something we want to rush).

What do you think?

Feb 10, 2012 at 11:26 PM

@drewmiller: agreed, we can punt that aspect for now and focus on excluding T4 packages from WMx

Feb 11, 2012 at 12:50 AM

Thanks @waltero and @davidebbo.

Adding a T4 capability sounds reasonable to me, too, especially considering the popularity of some of the packages that use it. This, then, will be the plan-of-record we'll use to start working on Monday (but folks, please do bring up any other angles we should consider if you think of them):

  • User-Agent is {0}/{1} ({2})
    • {0} = Client Name and Version, such as "WebMatrix 2.0 Beta" or "VS 10.0 SP1"
    • {1} = NuGet.Core version
    • {2} = OS Name/Version
    • This is possible today using NuGet.HttpUtility.SetUserAgent and NuGet.HttpUtility.CreateUserAgentString
  • Query string includes the "targetFramework" parameter (I'm not a fan of using the query string here, but since we're already doing it, we'd have to continue support it, so no real pressure to change it)
    • This is possible today using ISearchableRepository.Search
  • NuGet-Capabilities header:
    • Comma-separated list of capabilities the client can perform
    • Only two known values at this time are "PowerShell" and "T4"
    • When the client has no capabilities to claim, "None" can be specified as a "null" value, since the absence of this header indicates that no capability filtering/relevance should be performed.

Example requests:

WebMatrix

  • URL: http://nuget.org/api/v2/Search()?$top=5&searchTerm='jQuery'&targetFramework='net40'&includePrerelease=false HTTP/1.1
  • User-Agent: WebMatrix 2.0/1.6.21205.9031 (Microsoft Windows NT 6.1.7600.0)
  • NuGet-Capabilities: None

VS 10.0 SP1 Editing an ASP.NET MVC Application

  • URL: http://nuget.org/api/v2/Search()?$top=5&searchTerm='jQuery'&targetFramework='net40'&includePrerelease=false HTTP/1.1
  • User-Agent: VS 10.0 SP1 NuGet Add Package Dialog/1.6.21205.9031 (Microsoft Windows NT 6.1.7600.0)
  • NuGet-Capabilities: PowerShell, T4
Feb 20, 2012 at 6:21 PM

Hmm... My gut instinct was that this was a bad idea, but the more I think about it, the more I like it. As new capabilities surface, you get them by default, but then can choose to opt out later. 

Here's another idea too... When you your request to get the XML from the server that lists the 9 highlighted packages, that XML could also include the list of capabilities you want to filter out. Then, you don't even need to rev the client to filter a new capability out.

I propose the header name of NuGet-Capabilities-Exclude.  We could then leave the door open for either direction of filtering:

  1. NuGet-Capabilities-Exclude = Don't return any packages that require capabilities in this list
  2. NuGet-Capabilities-Include = Don't return any packages that require capabilities omitted from this list

Note: I'd like the NuGet-specific HTTP headers to be prefixed with "NuGet-"

 


 

From: waltero

We would like to make a small change to the proposal below. Instead of us simply passing: “NuGet-Capabilities: None” , we would like this to be something like ”Exclude-Nuget-Capabilities: PowerShell, T4”. The reason is that in the future if you decide to include a new capability such as “JavaScript” or whatever else. We will stop getting this nugets. With this change we explicitly say what we cannot handle of the capabilities you offer. Hope this does not cause much of a change on your side. 

Your thoughts? 

Thanks! 

Walter.

Feb 20, 2012 at 6:26 PM

Awesome! Thanks.

Walter.

Feb 20, 2012 at 6:27 PM

Before we call this a go though, I'd like to make sure Drew is in agreement too.

Feb 20, 2012 at 7:10 PM

First, let's be sure that we're on the same page regarding what the NuGet-Capabilities header is for. My understand is that it's for client features, not package contents or project-compatibility. If that's true, something like JavaScript would not be treated as a capability; that's about project-compatibility, and would be specified another way (we've talked about project types or framework types or languages types and so on; it's a much tougher problem than client capabilities and we need to spend some time designing it to get it right).

If we're talking about client capabilities, passing an exclude list has an issue, because you're opening yourself to packages that expect client capabilities you don't support. For instance, imagine we add a new client capability, assembly-based hooks for install, init, and uninstall, to the VS NuGet client. That's not going to be on your exclusion list, so until you update, you would get packages that expect the assembly hooks feature. They'd potentially be broken.

So, there are issues both ways: treating it as an include, and treating it as an exclude. You could have both, as @jeffhandley suggests, but that doesn't solve the problem of not being aware of new stuff that surfaces (and leaves the question: how would you reconcile specifying them both). That adds complexity, and I don't see any real value that it brings (but I'll admit I might be missing something).

I think for client capabilities, making the header a "this is what we support" is the best approach. That might not be the best approach for package contents and project-compatibility, but I don't think that's what we intend to use NuGet-Capabilities header for.

Thoughts?

Feb 20, 2012 at 8:28 PM

Sebastien Lambla brought up an interesting point on Twitter (https://twitter.com/#!/serialseb/status/171689810291982336), that some of the NuGet implementers might not be using NuGet.Core. Are we planning to use that part of the user agent for anything beyond analysis? What should a from-scratch implementer put in there? I'm thinking leaving it empty makes the most sense, but if we go that route, we need to make sure whatever parses this can handle that.

Feb 20, 2012 at 11:31 PM

There was a good point by @jeffhandley to introduce include/exclude parameters.

Thanks to @serialseb to notice about non- NuGet.Core based clients. In TeamCity we have pure-java implementation of NuGet feed. TeamCity feed does not export Search function in the OData feed.

It seems the filtering could be implemented in the sense of tags. Each package is tagged by a set of tags (i.e. T4, PowerShell, .NET40). Every client specifies a number of tags to include or to filter. In that sense if package is compatible to .NET20 and .NET40 the package could be tagged with both .NET20 and .NET40 to let client/sever to avoid to know any relations between tags.

How would those tags/properties computed for packages? Do you plan to update .nuspec file format to include it? It's likely to put all such attributes on package creation on order to simplify existing NuGet feed servers and to avoid code duplication between them.

Feb 21, 2012 at 12:58 AM

Nothing would rely on the NuGet.Core version being speified for this feature, or the capabilities being specified either. The filtering would only light up if the headers were supplied.

I'm really reluctant to overload the Tags field for this purpose, as I'm afraid of collisions for other purposes and making Tags so crowded that it becomes useless for actual tags.

What we've talked about offline is that we would calculate capabilities for packages and store it as additional metadata in the gallery, but package authors would also be able to specify the data to override our heuristics, either as part of the nuspec or just on the gallery.

Feb 21, 2012 at 9:32 AM

I did not meant existing package tags in my post. The idea was to introduce one more tag-like field for capabilities. 

I do not like the idea to make Gallery calculate package capabilities. All existing gallery implementations will be forced to re-implement it once more. As far as I know there are at least NuGet.org, MyGet, TeamCity, Artifactory, Nexus.

I with NuGet feed filtering capabilities will be included in package feed metadata.xml file to allow server declare if it supports such filtering.

Developer
Feb 21, 2012 at 5:15 PM

Chiming in late, sorry about that :-) If I look at the opening post, the main question here is that it should be possible to limit the view over a packages repository based on capabilities, right? Let's forget about the transport for a second (headers/additional OData filtering/...) and focus on that requirement: "limit the view based on capabilities".

Defining capabilities in my opinion is a tough one. What do you do with simple .NET library packages? What do you do with sample projects? What about MvcScaffolding? What about things that are supported only by WebMatrix? What about installing SignalR in Webmatrix? In short: I've seen a lot of concerns in previous posts in this thread and I'm in fact seeing a giant Carthesian product here: too many combinations possible. Too many edge cases!

The Windows Phone marketplace has a similar problem: how do you find what you want? I want all applications that have the "location" capability (use GPS). Try that search and see how many packages are coming back that actually don't use the GPS but where the application owner just checked every capability available to either show up in all searches or either because they don't know what these capabilities do. If users specify the capabilities used in their packages in some way, search will be broken. Then again: search is already broken since the repository contains unsupported packages for specific clients.

Question is: do we need it? Not to sound negative here, but... If my browser does not support HTML5, my Internet is broken. Fine by me! Bing should not include a filter to only search in sites that don't use Silverlight or Flash. If I support HTML but don't have JavaScript enabled, I'll get a degraded experience but I can still see if I should look into that website or not. Back to NuGet: If a package has a PowerShell script in it, WebMatrix does not run it. Simple as that. In some packages that's problematic, in others (like jQuery) that's not a problem. If I download T4Scaffolding, WebMatrix will barf. As a developer I know what T4 is and knwo I should probably try VS2010 to open this package. If all that <# #> jibberjabber is not appealing to me in WebMatrix, I simply uninstall the package and find another Facebook "like" button that does work in WebMatrix.

If you really need separate packages for separate products... Why not host a different gallery? Just like Orchard today has its own gallery, WebMatrix may have its own gallery as well.

Just my €0.02 (~= $0.026)

Feb 21, 2012 at 5:40 PM

Thanks for commenting @maartenba. I want to focus the conversation on the client capability part for now, specifically the PowerShell and T4 capability. I agree that if you start trying to add in package contents and project-compatibility, it's becomes a huge problem.

I did raise the idea of "do we really need to fix this; aren't our users smart enough to just uninstall if they install something that work work?" angle. The WebMatrix folks (@waltero) feel strongly about not having packages show up that won't have any chance that working, as the it targets a different developer audience than folks like you and me. A separate feed means curation, either by hand (at worst) or more likely by a tool that analyzes packages for selection into the curated feed from the nuget.org feed. That might make sense if it was just WebMatrix that was going to filter it. (@waltero, we talk about something like this; what where the other issues?)

Aside from the WebMatrix client, I've been toying with a MonoDevelop add-on, and there's nuget.exe itself. I do think it would be nice to filter out packages that require the PowerShell hooks. Although, in these cases, I think having them show up in the package list is okay, as long as they fail to install such that it's clear why they fail. On the other hand, if we can make good decisions about what can work and what cannot, should we show what can't work to the user by default? (I guess the cost associated with that *if* is the real issue).

About search being broken, that's a tough one. Ideally, package authors don't have to specify this stuff unless they want to override the automatic detection. We certainly don't want to make it "easy" to tweak it, for just that reason, As @eugenepetrenko pointed out, doing the detection on the server works great for us, but sucks for people writing their own feed. We could handle it during package creation in nuget.exe, but that has the problem of not updating all the packages currently on the gallery.

I'll give this some more though today, and wait to hear more from y'all, @jeffhandley, and @waltero before making my recommendation on whether and how we need to change the plan. We are, unfortunately, at a place where we have to lock down on this, at least for the first go of it.

Feb 21, 2012 at 6:34 PM

@jeffhandley's proposal is my preference:

  • NuGet-Capabilities-Exclude = Don't return any packages that require capabilities in this list
  • NuGet-Capabilities-Include = Don't return any packages that require capabilities omitted from this list

    @drewmiller's point that risks exist either way is valid, but I don't think they will come up very often and from looking at the possible set of releases of Webmatrix, we have space to accomodate any potential capabilities that could get added and cause problems. WebMatrix team is ready to close on it.

  • Developer
    Feb 21, 2012 at 8:55 PM

    Interesting discussion! I'm not a big fan of making clients too smart though (I mean the software, not the consumers.. well.. :))

    Trying to think outside of the box here, conceptually: what if (crazy idea) there was such thing as a virtual package (or what the heck, could be a chocolatey package as well :)) that describes a capability (which is actually a client requirement for the package to be used, in other words a dependency).

    As such, any NuGet package could specify a required capability by adding a dependency to such virtual package, optionally including a version (range).

    Example virtual packages:

    • PowerShell 2.0
    • PowerShell [1.0,2.0]
    • T4

    As such, packages could be filtered based on their dependencies, and if any unsupported pkg is detected, it is not listed. Analyzing this upon push, and storing it as readily present metadata next to those pkgs would improve performance; this could even be a means to gradually upgrade the nuget.org feed.

    Might make the nuspec/HTTP headers requirements a bit more lightweight.

     

    Feb 22, 2012 at 5:53 PM

    @xavierdecoster Yeah, I've had the same thought on the virtual dependencies.  But I took it one step further in my mind, where a package could declare something like:

    <ClientDependencies>
        <ClientDependency Id="PowerShell" Version="[1.0,2.0]" />
        <ClientDependency Id="T4" />
    </ClientDependencies>

    But as I read through all of these awesome comments, I wonder if a curated feed is really the only way to do this.  I have visions of:

    • http://nuget.org/api/v2/WebMatrix/Packages
    • http://nuget.org/api/v2/WindowsPhone/Packages

    Then on nuget.org new feeds could be created and managed.  During a design discussion on this topic, the idea of "NuGet.Proxy" came up, where NuGet.Proxy serves a feed, from source feeds, either filtering select packages out, or serving only select packages.  We wondered if this is encroaching too much upon what MyGet is already doing.

    Feb 22, 2012 at 6:29 PM

    @jeffhandley, I'm glad to see you landed back at a curated feed, because that's exactly where I landed, too, after the last round of comments.

    We've been talking this through on Jabbr (http://jabbr.net/#/rooms/nuget), and we think this is the best plan so far:

    • For WebMatrix (and potentially others as needed), there is a separate feed
    • This feed will be primarily managed automatically, based on defined rules (like, package has .ps1 files or has .t4 files, etc.)
    • Inevitably, the rules won't be enough, so there will be a way for an administrator to override the automatic curation on a per-package basis
    • Package authors can "elect" into the feed with tags (such as the ASPNETWEBPAGES tag that already exists, and is used by WMx 1)

    This keeps things simple, keeps things server-based (so we can tweak as needed), and causes no disruption in the NuGet ecosystem. It's also essentially what WMx is already doing, just with far better results in the filtered feed. It satisfies the WMx condition that lots of human management not be required.

    Note that we're not thinking of making this a "service" like MyGet, something that just anyone can use to build a machine-curated feed. In fact, we're not even sure how we would implement this (we've identified several approaches, but that's not important now), and whether it would be part of the gallery source or something separate.

    We would still like to tweak the user agent to add additional, useful information.

    I'll follow up with @waltero and the WMx folks on this, and make sure any additional comments make it back here.

    Developer
    Feb 22, 2012 at 7:10 PM
    Edited Feb 22, 2012 at 7:16 PM

    @drewmiller @jeffhandley I have a feeling we can help each other out here. We're working on a feature very similar to this. Would a MyGet feed work for the WM team? :-) Feel free to drop us an email at info@myget.org

    Feb 23, 2012 at 1:00 AM

    We've talked with @waltero and it looks like we're go for a feed-based solution to this, at least for now. We'll resume this discussion later for some of the other elements brought up (.NET Fx-filtering, for instance). Thanks all for helping us arrive here.

    @maartenba, I'll leave it to @jeffhandley and @waltero to consider whether MyGet would work, and what y'all have planned. Thanks for bringing it up, regardless.

    Developer
    Feb 23, 2012 at 7:09 AM

    Thanks! I've emailed @waltero about this.