Creating a ".NET Framework Packages" NuGet feed

Topics: Ecosystem, General
Coordinator
Feb 11, 2013 at 8:17 PM
As you have hopefully noticed, Microsoft is shipping more and more libraries as NuGet packages, which is wonderful! One downside to this is that when developers are seeking official ".NET Framework" components, it can be more difficult to find them. While it's great that community-produced packages are just as discoverable as Microsoft ones, this is hurting many developers. We have heard many comments that NuGet cannot be used in their enterprise because there's no quality/support guarantee like that of the .NET Framework and it's therefore an adoption blocker for libraries that ship exclusively on NuGet.

There's a proposal on the table that could both encourage more Microsoft teams to ship via NuGet and also alleviate the concerns about ".NET Framework" package discovery. The proposal is to create a ".NET Framework Packages" NuGet feed.

Existing Curated Feeds

The NuGet gallery at nuget.org presently serves a couple of "curated feeds" that Microsoft products use.
  1. WebMatrix has a curated feed that is used by that IDE to only show packages that are likely to work in ASP.NET Web Sites (vs. Web Applications).
  2. Visual Studio 2012 Express for Windows 8 uses a curated feed that only includes packages targeted at Windows 8 Store apps.
For each of those curated feeds, the consuming IDE explicitly opts into the curated feeds and gives the user the option to switch back to the full NuGet feed.

Proposed Curated Feed

The ".NET Framework Packages" proposal would create another curated feed that only includes packages that have been accepted into the ".NET Framework" (published by Microsoft, meeting the same quality and support bar as the .NET Framework proper). This curated feed would be available to all NuGet users, as an additional feed source that is registered by default much like the current "NuGet official package source" is. This feed source would be available from all Visual Studio SKUs.

Opting out of the feed source

The "NuGet official package source" cannot be deleted, but it can be disabled. The ".NET Framework Packages" package source would work the same way. By default, it would be enabled, giving all users 2 package source options: 1) NuGet official package source, and 2) .NET Framework Packages. Users that want to suppress the source could go into their package source options and disable ".NET Framework Packages."

Good, Bad, Indifferent

Good: There is concern from Microsoft product teams and customers alike that shipping via NuGet muddies the water of quality and support. How does one differentiate between a "Microsoft" package that was a side project for a feature team and has no quality or support promises, and a fully-supported, high-quality library that is worthy of being in the ".NET Framework" proper but was shipped via NuGet due to customer demand?

Bad: Microsoft-produced ".NET Framework" packages are given a level of differentiation or favoritism over community-produced packages, but only when the user specifically selects ".NET Framework Packages" as the only package source to browse/search.

Indifferent: Because this package source would only be added into your NuGet.config through the Visual Studio extension, it is fair to say that if you are a Visual Studio user then you are likely interested in discovering NuGet packages that are considered part of the ".NET Framework."

Soliciting Input

I think this is a terrific idea and a good approach to getting more Microsoft feature teams to ship their bits via NuGet, I want to hear your input. What would be your reaction to the ".NET Framework Packages" feed showing up by default when you install an update to the NuGet VS Extension?
Feb 11, 2013 at 8:28 PM
I'm a little confused at the original issue. Why would publishing .NET Framework packages as NuGet packages help adoption? Wouldn't you need the full .NET Framework (or the client version of) for say 4.5 to run .NET 4.5 applications?

Can you give an example of a scenario that is currently precluded and the proposal would help with?
Developer
Feb 11, 2013 at 8:46 PM
Edited Feb 11, 2013 at 10:09 PM
@wwahammy: This is about libraries that are considered part of the .NET Framework as far as quality, support, etc. go, but aren't actually included in the .NET Framework. Current examples of this include Immutable Collections, MEF, and TPL Dataflow.

A feed in NuGet for these libraries would make them easier to find and reference.

EDIT: To clarify, Immutable Collections is in a pre-release stage and so it doesn't currently have the same quality and support guarantees as the .NET Framework. It's an example of a library that would be a good candidate for this in the future.
Feb 11, 2013 at 8:53 PM
One way to get around the issue of favoritism would be for the ".NET Framework Packages" to not be a selected feed by default. The .NET Framework Packages would stay in the normal feed as well so by default there's no favoritism.
Coordinator
Feb 11, 2013 at 10:00 PM
@wwahammy The concern I have is discoverability of the subset feed. The users who need the filtered feed because their companies will only allow packages that fall within the ".NET Framework Packages" feed are often the same users that might not know how to find or add the additional feed. I would say the majority (or very large number) of Visual Studio users fall into this camp and having the default, out-of-the-box experience show you the subset feed is valuable for those users. On the other hand, users that don't care for the subset feed will likely understand how that it can be disabled.
Feb 11, 2013 at 11:08 PM
Well there are two issues here: First, if Microsoft wants to put it in their build of NuGet; they ultimately can. I think it gets really problematic to add it in because while it's positive to some users; it's also very positive to Microsoft who has the most control over the project. The other issue is why should Microsoft be the sole "trusted" feed? I mean there might be feeds of supported packages from other companies, how would they get added? In fact there might be companies that have even BETTER support that Microsoft would provide for their packages.

From a package creators perspective, it seem pretty unfair. They've had to work hard to get further up the list of packages based on popularity and then the second Microsoft has a new package, that package doesn't have to work as hard to be as discoverable.
Feb 12, 2013 at 6:41 AM
Well to be honest... while I can see a use case for this there are feature teams like signalr, aspnetwebstacknightly and tfs plugins which are Microsoft teams pushing their stuff to MyGet and pushing through stable packages to nuget.org.
Glimpse, nancyfx and dotspatial are non-Microsoft teams which are doing similar things. Bit puzzled why this wouldn't work for some other teams?
Feb 12, 2013 at 3:47 PM
@jeffhandley -> I'm ok for creating the additional feed for 'official .NET' packages.

This does bring up the question -- what about having another default-enabled feed for native packages? Or perhaps one specifically for CoApp-project produced packages (where we have a community maintaining the packages we build)?

My main concern is to ensure we get great discoverability for our packages (since they are by definition, the couple hundred most common OSS C/C++ libraries).

But it might be nice to have an additional, (still default) separate feed, which makes it easier to filter to that specific feed.

Really, once you start making additional feeds, you have to declare your criteria for having that feed included by default, and allow anyone to meet that criteria.

In the CoApp project, we had a thought of a feed-of-feeds that the package manager would reference for this purpose. We haven't yet done that tho'.

G
Feb 12, 2013 at 4:35 PM
I like how @fearthecowboy is looking at this. The community should decide on objective criteria for what makes a feed enabled by default. After that's done, there's far less debate over whether any group or company has their feed enabled by default; either the feed meets the criteria or it doesn't.
Developer
Feb 12, 2013 at 5:58 PM
@wwahammy, maartenba, and fearthecowboy: Thanks a lot for the honest feedback.

Let me clarify a few things and outline the goals we pursue with this.

First of all, our intention is not promoting Microsoft packages over non-Microsoft packages. Our intention is separating the platform from platform extensions. The platform is delivered by Microsoft whereas platform extensions are delivered both by Microsoft and by 3rd parties.

Think of the existing Add Reference experience in VS. It shows a list of .NET Framework assemblies. In the future, we envision that less and less functionality will ship "in-box" and more and more functionality ships in an out-of-band fashion. Good examples are the new version of MEF and TPL Dataflow or ASP.NET MVC. In the case of MEF and TPL Dataflow we actually considered adding them to the in-box version of .NET but decided against it so that we can update those packages more frequently.

We strongly believe that moving to an out-of-band deployment model is more beneficial to the .NET ecosystem. However, as Jeff pointed out, we are receiving the feedback that OOB platform components aren't as discoverable as in-box platform components. The current proposal Jeff outlined above tries to address that concern.

Thoughts?
Feb 12, 2013 at 6:24 PM
@terrajobst -> I know that wwahammy and I both understand that completely.

(fyi, I work for Microsoft, in the Open Source Technology Center)

The point of significance is that my work has virtually same concerns--we're producing native library packages and want to ensure that they are very discoverable as well as anything else. Either I stuff those into the common NuGet repository, or I produce and manage my own repository.

And you know what? I'm betting that there are other organizations that have the same concerns too.

So, rather than solving the problem in a one-off manner, which would have the side effect of appearing to favor Microsoft individually, wwahammy and I are proposing that there be a policy that describes how any organization can get their feed of packages included in the client by default.

That way both you and I (and any other organization that meets the standards set down) gets treated identically.

The Outercurve Foundation can help manage the policy process, and everybody wins.
Feb 12, 2013 at 6:41 PM
<div style="font-size:11pt; font-family:Calibri,sans-serif">Big plus one!<br> <br> Sent from my Windows Phone</div> <hr> <span style="font-weight:bold; font-size:10pt; font-family:Tahoma,sans-serif">From:
</span><span style="font-size:10pt; font-family:Tahoma,sans-serif">fearthecowboy</span><br>
<span style="font-weight:bold; font-size:10pt; font-family:Tahoma,sans-serif">Sent:
</span><span style="font-size:10pt; font-family:Tahoma,sans-serif">dinsdag 12 februari 2013 20:24</span><br>
<span style="font-weight:bold; font-size:10pt; font-family:Tahoma,sans-serif">To:
</span><span style="font-size:10pt; font-family:Tahoma,sans-serif">maarten@balliauw.be
style="font-weight:bold; font-size:10pt; font-family:Tahoma,sans-serif">Subject:
</span><span style="font-size:10pt; font-family:Tahoma,sans-serif">Re: Creating a ".NET Framework Packages" NuGet feed [nuget:432758]</span><br>
<br>
<p>From: fearthecowboy</p> <div id="ThreadNotificationPostBody">@terrajobst -&gt; I know that wwahammy and I both understand that completely. <br> <br> (fyi, I work for Microsoft, in the Open Source Technology Center) <br> <br> The point of significance is that my work has virtually same concerns--we're producing native library packages and want to ensure that they are very discoverable as well as anything else. Either I stuff those into the common NuGet repository, or I produce and manage my own repository. <br> <br> And you know what? I'm betting that there are other organizations that have the same concerns too. <br> <br> So, rather than solving the problem in a <strong>one-off</strong> manner, which would have the side effect of <em>appearing</em> to favor Microsoft individually, wwahammy and I are proposing that there be a policy that describes how <strong>any</strong> organization can get their feed of packages included in the client by default. <br> <br> That way both you and I (and any other organization that meets the standards set down) gets treated identically. <br> <br> The Outercurve Foundation can help manage the policy process, and everybody wins.<br> </div> <div></div>
Feb 12, 2013 at 6:54 PM
For those who wondered what maartenba said:
"Big plus one"
LOL, perhaps a bug should be filed for Codeplex :D
Developer
Feb 12, 2013 at 8:58 PM
fearthecowboy wrote:
So, rather than solving the problem in a one-off manner, which would have the side effect
of appearing to favor Microsoft individually, wwahammy and I are proposing that there be
a policy that describes how any organization can get their feed of packages included in
the client by default.
Honestly, I'm not sure this would end up helping our customers. We need to keep in mind that Visual Studio is used for a large variety of developers and projects. I don't think offering a process where any organization can install default feeds is super helpful. I believe less is more in that context.

Our existing customers told us time and again they value the stability and feature richness of the existing .NET Framework. My team doesn't want to change that. Over time, our goal is to slowly move this everything-gets-installed-by-default-model into an out-of-band model. However, in order not to lose existing customers we need to be very careful that the platform functionality remains discoverable.

I understand that you could see that as NuGet favoring platform functionality. I see the reverse: platform functionality favors the NuGet deployment model. Granted, this comes at the expense of one default entry being added during the installation of our product.
The Outercurve Foundation can help manage the policy process, and everybody wins.
My thinking assumes NuGet's primary audience is managed developers. Reading you comments, it seems you are in a different world, i.e. native. The package manager dialog could probably take the current project into consideration, which means the registered feed would need to be language/platform specific.
Feb 12, 2013 at 9:23 PM
Edited Feb 12, 2013 at 9:24 PM
TerraJobst: I appreciate your concerns about your customers. It's important to appreciate as well that Microsoft is one member of a very diverse NuGet community, albeit a very important member. Whether intended or not, adding this into the open source NuGet codebase implies that one particular member of the community should be favored over other members which discourages other members of the community. If package creators don't feel they're on a level playing field, they may not be as willing to participate. Fewer packages and fewer pull requests from developers decreases the value of NuGet for everyone.
Feb 12, 2013 at 9:33 PM
Yes, currently NuGet is a predominantly for managed developers.

I didn't spend the last several months working with the NuGet team and Visual Studio to come up with a universally flexible model to support NATIVE and MANAGED libraries in a frictionless manner to end up a second class citizen in a universe where the amount of native code outweighs managed code by several orders of magnitude.

Taking the attitude that Microsoft above all (which is implied when Microsoft gets to have it's own default repository, a luxury not afforded anyone else) certainly doesn't place a lot of faith in the meritocratic model of open source.

Especially since Microsoft doesn't own NuGet; it's an open source project in the Outercurve Foundation.

And having multiple default feeds doesn't hamper discoverability; it greatly enhances it, since with a simple click on a dropdown you can filter down to a single feed, having multiple repositories installed is FAR, FAR preferable than having every package in the universe dumped into the same repository (the official nuget.org repository).

Because if C/C++ developers can't see CoApp packages by default in a separate feed, I'll just insert them into the default NuGet repository, and we're back to one huge collection.

Visual Studio caters to an awful lot more than .NET customers, and it would be handy if you'd remember that.

Visual C++ is counting on this for native libraries just as much as .NET does.
Editor
Feb 12, 2013 at 10:11 PM
This seems too complex. Here's my counter proposal. It's more complex but it should solve everyone's issue.

Users

There should be users you can trust. Microsoft stuff should use the Microsoft user. Telerik stuff should be the Telerik user. If some dude inside Microsoft somehow published some side thing with the name "Microsoft," just change the name to his. It's not cool as [Organization] needs to mean something

Urls

URLs are too complex. Easy fix.

Users

  • nuget.org/shanselman would be me
  • nuget.org/microsoft would be Microsoft
You get the idea.

User's lists.

Users should have lists just like Twitter. So, Microsoft user logs in, makes a BCL list. They can add (pin) packages to the list.

So, then nuget.org/microsoft/bcl is a list as is, perhaps nuget.org/shanselman/favorites

Prereqs are pulled from the list first, and fallback to the default feed if not found.

Feed Discovery

If I type in nuget.org/shanselman/favorites into a feed thing in a NuGet dialog box, it autodiscovers the feed via an autodiscovery link in the HTML.

<link rel="nuget" type="application/atom+xml" title="Scott Hanselman's NuGet favs" href="whatever" />

This would mean I could even make hanselman.com a valid alias for my favorite NuGet packages.

This would solve everyone's issue, IMHO.
Feb 12, 2013 at 10:19 PM
That works too.

Frankly, I don't care about implementation as much as I don't want to see any organization get granted special 'enabled-by-default' powers that are not available to any other organization (passing whatever set of arbitrary criteria you want to be worthy of such value.)

G
Editor
Feb 12, 2013 at 10:28 PM
Edited Feb 12, 2013 at 10:28 PM
I'm pretty sure my idea means everyone is the same.
Feb 12, 2013 at 10:55 PM
Edited Feb 12, 2013 at 10:56 PM
Hey Jeff, --

I'm excited that you're looking to make changes to Nuget to increase "official" package discoverability! The idea of shipping (and maintaining in the future) a default set of feeds should definitely be helpful for Nuget users looking to separate the wheat from the chaff. (Last year, I demonstrated that official packages in the main feed can easily be confused with malicious fakes or lower-quality versions.)

But I think you need to tread carefully here. Like wwahammy pointed out, Microsoft is just one of many players in the Nuget space. A change to Nuget feed defaults catering specifically to Microsoft would alienate other organizations with similar (if not identical) discoverability issues. I don't think managed vs. native is really relevant to this discussion.

I think fearthecowboy has good points -- if we allow changes to feed defaults, the process needs to be documented and opened up for other organizations. But I do recognize that this process needs to be sufficiently stringent to ensure this default feed list doesn't grow unwieldy. (Alternatively, perhaps instead of feed changes, Nuget can implement changes to its metadata/UI offering "filter by organization name"?)

Look forward to continuing the discussion!

/rafael
Feb 12, 2013 at 11:01 PM
I'm a C/C++ software developer and I'm a little bit concerned reading this proposal.

I have been observing and cheering up the CoApp project with huge hope it will solve tons of annoying problems for nations working with Visual Studio and Windows platform regarding deployment and use of third-party software dependencies like open source software with less or even no pain.
CoApp aims to solve problems that no other project even dares to try and does it using Microsoft's own technologies that most of software developers on Windows are scare to even think about, not to mention reaching snippy documentation on MSDN.

In fact, IMHO, there has been already huge amount of dedication by Microsoft to make life of .NET developers easier, whereas areas important for deployment and interoperability in the native world have been neglected. I see CoApp as a Microsoft's heaven-sent initiative.

Having said that, I don't want to see CoApp being moved into the second category citizen.
If NuGet becomes a default software feed for Microsoft, then existence and future of CoApp project might become debatable.
That may lead to lack of motivation and support from decision makers for further CoApp development, what would be a crying shame.

"My project for Microsoft" exclusive deals are bad for all of us.
Editor
Feb 12, 2013 at 11:05 PM
Hey, any comments on my proposal?
Coordinator
Feb 12, 2013 at 11:12 PM
3 new messages came in while I was writing this, and I haven't read them yet... I will update again ASAP.

@shanselman - Yes, wonderful ideas! This is what we want. I admit that it's hard to balance feature requirements for NuGet itself, keeping the gallery running smoothly, and implementing the new capabilities to have N feeds. We're trying though, and we are solidifying the API ideas which cover exactly what you've described.

For reference, here's the working notes on what the next NuGet API would provide:
https://github.com/howarddierking/NuGetGallery/wiki/Api-v3-Overview

That list includes A) packages by author, B) packages by owner (or organization), C) list of alternate feed sources D) packages followed by a user. Our current discussions on following packages include the ability to "tag" a package when you follow it, automatically generating new feeds for each unique tag used when following packages.

We will arrive at a distinction between a "Gallery" and a "Feed" where a Gallery serves N feeds. Any user can create N feeds for themselves. Then by providing VS with your account all of the feeds you have created on the gallery automatically show up as being available to you (you don't even have to register new feeds). Of course, MyGet (or any other gallery) will be able to provide the same feature too, where when I add "MyGet" into my list of registered "Galleries," it will be able to serve up the list of feeds that I have on that gallery.

So, in the end, the "dotnetframework" or "microsoft" or "microsoft/bcl" feed, however it manifests, is just one of countless feeds served by the gallery. But that circles back to the issue at hand, which @fearthecowboy has highlighted: does this one feed deserve to be enabled-by-default?
Feb 12, 2013 at 11:14 PM
jeffhandley wrote:
So, in the end, the "dotnetframework" or "microsoft" or "microsoft/bcl" feed,
however it manifests, is just one of countless feeds served by the gallery.
But that circles back to the issue at hand, which @fearthecowboy has highlighted:
does this one feed deserve to be enabled-by-default?
No.
Editor
Feb 12, 2013 at 11:17 PM
Edited Feb 13, 2013 at 12:38 AM
I believe that the existing “firehose” feed is the default feed. Anyone can get in there, as it is today. Folks can then “turn down the volume” by removing it and using x number of curated feeds.
Feb 12, 2013 at 11:18 PM
shanselman wrote:
Hey, any comments on my proposal?
Well, I did say "that works too..."

While your proposal effectively handles filtering by source (ie, publisher) (and for my part, looks good) does that still mean that the BCL team would publish to a nuget.org hosted repository? Do they want that, or do they want that server to be in their actual possession (like Microsoft tends to want to do).

If it's in their possession, then the client needs to be able to know about it, which means that it has to get published in a known location.

Does that get registered at nuget.org so when someone uses nuget they see those packages by default? Does Apache or Outercurve or CoApp get the same abilities? How about Telerik?

When the dust settles, I'd prefer a model that supports:
 
"Publishers that qualify with <some fair standards for publishers>, will have their feed searched when people look for stuff."
 
And then let NuGet provide convenient filters on top of that (by whatever means and criteria the user cares about).
Feb 12, 2013 at 11:21 PM
fearthecowboy wrote:
When the dust settles, I'd prefer a model that supports:
 
"Publishers that qualify with <some fair standards for publishers>, will have their feed searched when people look for stuff."
 
Amen.
Feb 12, 2013 at 11:34 PM
shanselman wrote:
Hey, any comments on my proposal?
Hey Scott! I had my editor open for quite some time, so didn't see your proposal come in.

I agree with your idea of establishing a concept of "trusted" users or organizations. (Nuget has fundamental package trust issues.) I like your feed construction templates/formats but I think the root issue being discussed right now is the inclusion of an additional Microsoft-specific feed in the default list of feeds delivered as a part of Nuget "global".

I think the proposed idea here is to offer users a distinction between official and normal packages without additional feed knowledge. (The addition of external Nuget feeds is possible today, but discoverability is terrible.)

Did I capture your thoughts correctly?
Coordinator
Feb 12, 2013 at 11:40 PM
Having criteria that anyone can meet to make it into an advertised set of feeds makes sense. We can define those criteria as a community. At the same time, I would like the community to help the core team solve the problem of customers that are currently afraid of NuGet as the default or sole shipping vehicle for Microsoft libraries. How do we ease their fears/restrictions while maintaining an open system where Microsoft does not get some sort of favoritism?
Feb 12, 2013 at 11:48 PM
jeffhandley wrote:
We can define those criteria as a community.
Excellent.
How do we ease their fears/restrictions while maintaining an open system where Microsoft does not get some sort of favoritism?
Have those community standards apply to Microsoft's feed, and make it trivial to filter to that feed.

How does that not address their fears? Just how much irrational fear can they have?
Feb 12, 2013 at 11:51 PM
I don't think the issue is whether there's a use case for the Microsoft feed being on by default. The issue is whether an open source, community run project should preference one member of the community over others.
Feb 12, 2013 at 11:58 PM
Edited Feb 12, 2013 at 11:59 PM
jeffhandley wrote:
At the same time, I would like the community to help the core team solve the problem of customers that are currently afraid of NuGet as the default or sole shipping vehicle for Microsoft libraries. How do we ease their fears/restrictions while maintaining an open system where Microsoft does not get some sort of favoritism?
I think what has been proposed by others so far resolve the discoverability problem shared by Microsoft and other organizations. Do they not agree? What don't they like?
Editor
Feb 13, 2013 at 12:00 AM
Edited Feb 13, 2013 at 12:38 AM
I say the search searches the firehose.
Feb 13, 2013 at 12:04 AM
@jeffhandley: As with the case of any library creator, it's the responsibility of the creator to recruit users.

One idea that comes to mind though is the idea of using a Windows policy. A domain administrator could set a list of "approved" feeds and that could be propagated out to all computers including developer ones. This could also be applied at the individual system level the same way. NuGet could look whether the approved feeds policy is set and use those feeds if they are.
Feb 13, 2013 at 12:08 AM
@shanselman: I've gathered that at least some teams at Microsoft want to host their own feed instead of using the NuGet.org one. I like the search idea better though personally but it may not be feasible.
Feb 13, 2013 at 2:33 AM
wwahammy wrote:
One idea that comes to mind though is the idea of using a Windows policy. A domain administrator could set a list of "approved" feeds and that could be propagated out to all computers including developer ones. This could also be applied at the individual system level the same way. NuGet could look whether the approved feeds policy is set and use those feeds if they are.
Another possible approach is to give some control of the default feeds to the IDE. For example,
  • NuGet's VSIX and command line tool continue to ship only the default feed (NuGet.org).
  • Microsoft can add ".NET Framework Packages" feed to Visual Studio NuGet's VSIX's feeds (probably via a configuration API exposed by NuGet), so that Microsoft customers can discover OOB packages more easily. As Visual Studio is Microsoft's own product, allowing Microsoft to modify/override the default feeds won't hurt NuGet's neutrality.
  • Other IDEs, such as SharpDevelop/MonoDevelop, can also customize the feeds NuGet used, when NuGet is hosted by them.
  • Organizations such as Telerik can also add their own feeds by calling the same API in their VS add-ins (or SharpDevelop/MonoDevelop add-ins).
  • The users can also manually add their feeds to meet their own needs, just as they do today.
http://nuget.org/packages is just like the Apple's App Store. It is a fair place for all packagers, but the governors should take some actions recently to clean it up.
Developer
Feb 13, 2013 at 4:36 AM
(I work for Microsoft on .NET Framework team)

+1 for @lextm's suggestion. NuGet has zero knowledge about specific feeds other than its own, each product that ships with NuGet can register their own default feeds.
Feb 13, 2013 at 5:09 AM
Just to be complete: at MyGet we allow feed owners to aggregate other feeds (optionally filtered) into a single feed, using underlaying configurable pkg sources for that feed.

This provides server-side options to tweak the contents of the feed (to feed owner) without the need for consumers to modify nuget settings. This obviously implies a level of trust between consumer and feed owner, which is no different from nuget.org right now. Also, we didn't start from a "one feed to rule them all" concept, which likely makes this more relevant. I think this feed owner maps pretty much with the Organization that should mean something, as suggested by shanselman. Feed owner is an important concept which is not really a first-class citizen on the NuGet Gallery.

This is a fundamental difference in approach & design.
Coordinator
Feb 13, 2013 at 5:23 AM
I too like @lextm's suggestion.

The NuGet VSIX and nuget.exe will:
  1. Have the default "Official NuGet Package Source" feed that it has now, which is hard-coded into NuGet itself.
  2. Read a registry key for additional feeds
  3. Read a registry key that can suppress the "Official NuGet Package Source" (for the ultra conservative companies)
  4. Read a registry key that can prevent additional feeds from being configured in nuget.config files
  5. Read nuget.config files to allow registry-specified feeds to be disabled
  6. Unless blocked by #4, read nuget.config files to allow additional feeds to be configured
The nuget.org gallery will:
  1. We will continue to design and then implement the new API with the goals of N feeds coming from the gallery, including feeds by owner, and feeds for followed packages (with optional tagging)
  2. When #1 is complete, this will yield a feed that Microsoft controls by way of package ownership. But any other person or organization can also publicize their own feed as well.
  3. In the meantime, create a one-off curated feed for "dotnetframework" much like what was done for WebMatrix and VS Express for Windows 8. This implementation will die off as soon as the new API is in place though.
Visual Studio will:
  1. Write to the registry to configure the ".NET Framework Packages" feed that becomes an additional feed visible by default in VS. The Official NuGet Package Source feed would still be available too, of course.
  2. Use a FWLink that points to the curated feed, allowing that URL to change to the new API when it's ready.
  3. Guide customers that need to operate in a locked-down manner to use group policy to further control or constrain the feeds available in VS.
Is this an agreeable balance where NuGet enables other products and companies to add/remove feeds, while not granting favoritism to any party? It also side-steps the whole concept of determining what a "trusted" feed is. If we determine that is a useful feature in its own right, then we could explore it.

I appreciate the input on this.
Feb 13, 2013 at 5:24 AM
For the record, that also includes aggregating external feeds. I forgot to mention that.
From a user/organization perspective:
  • nuget.org is a large package repository where you can publish pkgs
  • myget.org is a large feed repository where you can create/aggregate feeds and publish pkgs to your feeds and the external ones if you have their API key.
Note: feed owners can set granular permissions on each feed they own.
Coordinator
Feb 13, 2013 at 5:51 AM
The MyGet stuff makes sense, @xavierdecoster. I know there will be some overlap when nuget.org starts to serve N feeds and it is easy to have a feed-by-owner, but I also think the roles of nuget.org and myget.org are quite different as you are pointing out.

This thread isn't the right place to go into details about the new NuGet API, but we will be investing a lot more energy into that very soon. When we do, I'll make sure we connect with you so that MyGet scenarios are accounted for in the API as well.
Developer
Feb 13, 2013 at 8:36 AM
2.Use a FWLink that points to the curated feed, allowing that URL to change to the new API when it's ready.
If possible, I recommend not using a FWLink. It causes one extra request and we had several issues with the FWLink in the past with the NuGet feed.
Developer
Feb 13, 2013 at 1:17 PM
@jeffhandley: This proposal would work us. Ultimately, the only thing we care about is the experience a developer gets when installing VS on a fresh machine. To me it doesn't matter which tool is responsible for which piece. However, I fully understand that there is a fine line between what NuGet can do and what it shouldn't.

Thanks!
Feb 13, 2013 at 5:11 PM
Edited Feb 13, 2013 at 5:11 PM
@jeffhandley: As for the NuGet source code, I don't believe there is any reason why the changes would harm neutrality.

I am concerned though about NuGet.org, a community run site, having "one-off" feeds for only one community member even if it's temporary. I think for the long term neutrality of NuGet it would be best for the "one-off" feed not be on NuGet.org. (In fact I'm not sure the current one-off feeds should even be there for that reason either.)
Developer
Feb 13, 2013 at 6:51 PM
Edited Feb 13, 2013 at 6:53 PM
I see favoritism as a non-issue, as long as MS does things in a way that anybody else could do them if they owned the development tool in question.

a) the 'favoritism' really happens in Visual Studio configuration. Which Microsoft owns.
b) nothing else that happens is special to Microsoft or out of anybody else's reach

For instance, anyone could take the gallery source, deploy their own nuget gallery, call it 'nuget.mycompany.com'.
But it would be great if its easy for anyone to create their own organization's 'official package feed' on nuget.org, instead of rehosting the gallery.
Coordinator
Feb 13, 2013 at 10:43 PM
For the record, I too am not happy with these one-off curated feeds and I want them to yield to the better approach of owner-based feeds coming from the gallery for any package owner. Then Telerik, Apache, or anyone else could say "if you want to quickly find our packages, then follow the 'nuget.org/feeds/telerik' feed."

The 2 existing feeds (and the 3rd we're discussing here) are simply stop-gap measures. In order for NuGet to become a first-class experience within WebMatrix, it was a requirement to have the WebMatrix curated feed. In order for NuGet to be shipped in VS 2012 Express for Windows 8, it was a requirement to have the Windows 8 curated feed. The same request is now coming through for more ".NET Framework" packages to be shipped, that VS needs to be able to give customers a curated feed.

We will work on the owner-based feeds but I hope the community can accept that the stop-gap one-off curated feeds were an excellent way to drive NuGet into ubiquity. Without them, we would not have been able to accomplish NuGet being included in every SKU of VS 2012.

I also want to say that there is a very important detail about how the curated feeds work; these are very simply subset feeds. They are absolutely not separate "galleries" of packages. The default feed has all packages on nuget.org, and these curated and owner-based feeds are merely subsets of the entire gallery. MyGet serves the need when package owners need to put their packages up on isolated, separated sets of packages. For instance, the ASP.NET Web Stack nightly builds certainly don't make sense on nuget.org where any user could stumble upon them and accidentally start using daily builds. That is why nightly builds go to MyGet - it's an extremely explicit, informed decision to start installing those builds.

While it's true that anyone could clone the gallery and spin up their own deployment, I hope that doesn't happen beyond scenarios like what MyGet already addresses. I think it's crucial to the success of NuGet for nuget.org to be the home of publicly released packages that should reach the mainstream.
Developer
Feb 14, 2013 at 6:35 AM
@Jeff you're right, galleries are really different from feeds. Hosting galleries doesn't make much sense for anyone if they want their package to be able to be part of a unified dependency ecosystem.
Feb 18, 2013 at 5:41 AM
With regards to the original idea, I actually think Scott's suggestion makes most sense: just have /users/maarten and list that author's pckages. THis doesn't need policies or procedures, it'll "just work".
Coordinator
Feb 19, 2013 at 10:41 PM
I really like Hanselman's suggestions. It's inline with my thoughts on NuGet security that I've talked about for a long time and finally gotten around to writing down in a blog post. http://haacked.com/archive/2013/02/19/trust-and-nuget.aspx

The key part is being able to establish identities and having feeds for every identity. That's what we should focus on with NuGet first before adding more and more curated feeds.

Ideally, a curated feed is really just a verified user feed.

I definitely don't like the idea of any curated feeds being in the NuGet source by default. Definitely make that a generic feature that Microsoft's installer so happens to set up (as I've seen proposed here).
Feb 21, 2013 at 3:47 PM
Why you need official .NET Framework components as Microsoft is shipping them unofficially?

NetUltimate
Coordinator
Feb 21, 2013 at 4:37 PM
@netultimate7 Microsoft customers, including many at the MVP summit this week, have said they need a subset feed for Microsoft packages that are the official packages released by Microsoft. And Microsoft releases a lot of software officially, and even exclusively, on NuGet. But as stated herein, Microsoft is not alone in this requirement.
Developer
Feb 23, 2013 at 7:27 AM
Jeff, it's a spam bot. Ignore it.
Mar 18, 2013 at 8:24 PM
Edited Mar 18, 2013 at 8:39 PM
It's been quiet on the package source discovery front, but not anymore. We've released a draft discovery specification at https://github.com/myget/PackageSourceDiscovery, a specification Xavier, Scott, Phil, Jeff and myself have been working on.

We've also put online two posts on the topic:
Comments are welcomed.
Mar 20, 2013 at 2:09 AM
@Maartenba Hey Maarten, good to catch up with you a little at summit. I like the idea here and the approach of discovery, but the one thing I'm not clear on is how to aggregate the feeds into something that is discoverable. What you have here is perfect if I know there is a feed in your rel="Nuget" on http://blog.maartenballiauw.be, but what if I don't? I guess what I'm wondering is: where is the meta-search? I want to search the feed of feeds for feeds from Microsoft (or Telerik, or whomever, or maybe from the MyGet Gallery).

@shanselman +1 to the idea of user lists. Could this also be used to help land packages in the correct places where it might not be clear as to which one should be used? Like, if the shanselman/AspNetMvc feed had the Ninject.MVC3 package in it, but perhaps not the Ninject base package, for instance, you wouldn't try to install Ninject and then wonder why you're getting build errors on Bootstrapper missing. This could allow the AspNetMvc feed to be "cleaner" so that devs who don't know the correct package to install won't have as much trouble (though, I suppose cross-feed dependency resolution gets tricky too here). Don't know if that's the right approach, but it's a thought.

I'm most in favor of Scott's approach, and like the discoverability that Maarten et al have worked out. Going that route - where anyone can curate a feed - gives me the ability to trust whomever I like. I just also want to see search exist over the "feed of feeds".
Mar 21, 2013 at 3:13 PM
I've been chatting with @Maartenba about the Feed Discovery for a little while now (for the Nuget Extensions Project, and thought I'd share our team's comment with regards to the "crawling" idea: we are not aware of any protocol that does crawl-discovery in this manner. This is also problematic, not just for dealing with circular references and mixed status-code (several 200’s, one 500), but in authentication. It’d be insecure to pass the same credentials to different URLS, yet awkward to continue prompting for every 401.

As for @shanselman's idea to do a web request to search for a <link> element in the DOM of an HTML page... this is asking for a lot of complexity. For example, how could an implementer tell if a string is HTML? DTD validation could be used, but many websites do not have valid HTML. What if JavaScript is used to add this to the page page (as it often is)? Virtually all consumers of HTML treat that as valid as well. This complexity would lead implementers to use different methods for dealing with HTML, and thus to an ecosystem where each implementer works slightly differently.

Both ProGet Client Tools and ProGet are implementing the NuGet Feed Discovery Spec, which were pretty easy to implement and consume... especially since we like keeping our code incredibly simple.
Mar 21, 2013 at 3:20 PM
@MisterJames it's like with blogs. If you use my URL and there is an RSS link on there, your reader can use it. If not, well... no RSS then :-) Perhaps we need some central place where one user can list all their feeds using discovery.
Mar 21, 2013 at 3:34 PM
@MisterJames @maartenba A centralized "feed of feeds" (i.e. a searchable list of searchable lists of packages) is pretty pointless because there's already a centralized package source (NuGet.org). If you want to pick and choose the feeds you want to consume, just use a private repository (ProGet, MyGet, Artifactory, Nexus) -- they all support server-side aggregation.
Mar 21, 2013 at 4:32 PM
@apapadimoulis But we're not talking "discoverability" here yet, we're just talking about "stumblability". I agree with you on the point of centralized feeds, but I'm not suggesting that and don't propose scrubbing/scraping sites. I'm just noting an application of what @shanselman and other had suggested in regards to having someuser/feedname.

The feed of feeds search has relevance. I want to look for a package "under" this "user", or I am a corporate user that is finally allowed to use NuGet, but have a restriction that says "you can only use Microsoft curated feeds". Unless you require the feed curator to also maintain an "all packages" feed (which is an option) I would have to search in each of their feeds separately.

More simply stated, what I'm saying is that if the idea of lists is built out, I'd like to be able to search at the level of the user or the list at my discretion. Automating a someuser/all feed would likely take care of this use case.
Coordinator
Mar 22, 2013 at 6:37 AM
@MisterJames, I think we are working on a feature that will address what you're asking for. We're going to let you follow packages on nuget.org, and you'll also be able to follow package owners, as well as follow the packages/owners a user follows.

For instance, you could follow the "dotnetframework" user's packages, as well as others' packages. Then, all of your followed packages becomes a feed for you, automatically. When you follow a package, owner, or user, you can optionally apply a tag. Any tags you follow packages with will become another feed for you.

These are scenarios we're designing into our new API. We recently created a new repo to house the specs an everything for that API, and we're building the materials out in there. https://github.com/nuget/nugetapi

Thoughts on this?
Mar 22, 2013 at 11:09 AM
@JeffHandley What's the best way to provide specific feedback on the new API ideas?

In general, we'd like to see the new API give first-class support/consideration for private repositories, instead of requiring us vendors to work around the design. Private repos are an absolute must for enterprise development, but things like organizations, announcements, and reporting abuse simply don't fit into the private repo model. They're obviously really important for nuget.org, so separating the API into "repository" and "NuGet.org Gallery" endpoints (or whatever) would be ideal.

Would you be open to that? If so we can contribute to the "repository" specs, since that's where our expertise lies.
Mar 22, 2013 at 2:45 PM
@apapadimoulis, as I work in a big firm, I do see such things like organizations, announcements, and reporting abuse useful, as they allow different departments to take care of their own packages. They might not fit into small and medium enterprise scenarios, but in those cases they should be set as optional. I don't think they should be treated as secondary.
Mar 22, 2013 at 3:22 PM
@lextm at first glance they may seem useful -- and even tempting to try -- but having been worked in the space (through Inedo) at enterprises of all sizes, they most certainly enable worst practices and should never be included in any private repository. Let's pick an easy one: reporting abuse.

On nuget.org, it's used for abusive packages: dangerous, offensive, illegal, etc. This is pretty important considering there's no vetting process for getting packages published.

In an organization, if you have an abusive package then you have a MUCH BIGGER problem, and this should be reported through different channels a "report abuse" link. Let's say you discovered a package (written by a rouge employee) that posts sensitive customer data to some server via a sneaky property accessor. Would you, as a developer, seriously feel that you've done your due diligence by clicking a link and filling out a form on your internal nuget server? Would you, as the dev manager, want your developers to advise him of such breaches like this? Obviously not, so that means this "report abuse" feature can only be misused. Are you going to submit bugs with it? Are you going to do code review with it? Any use for it is inappropriate by default (it's not reporting abuse), it's just a question of what type of screw you're going to use this hammer on.

The same is true for many of the nuget.org requirements. If some enterprise is naive enough to think that they should have an internal version of nuget.org, then they can do just that. I've seen it in the field, and it works incredibly poorly. Most will just settle on an actual private repo.

The "repository" spec is what nuget.exe (or vs client) needs to talk to a server about search, download, upload; the rest can be handled on the web ui or via an implementation-specific server api (like the ProGet API).
Coordinator
Mar 22, 2013 at 4:53 PM
For discussions about the new NuGet API design, you can read the spec here:
https://github.com/NuGet/NuGetApi/blob/master/specs/API-v3-Specification.md

And then you can file issues on that github repository.

Private repositories are out of scope for nuget.org. MyGet.org does a good job with private repos, and we don't intend to go that direction with nuget.org. Instead, nuget.org will be the home of public packages, but we'll be providing several means for serving subset feeds. Think of it this way: nuget.org is the gallery of all public packages, and it will serve customized feeds that are all subsets of the full gallery. No feed from nuget.org will contain a package that is only available on that feed.

We do understand the important of companies needing to have private feeds. For that, there are several options:
  1. MyGet.org
  2. File share based feeds
  3. NuGet.Server based feeds
  4. Internal deployments of the NuGet Gallery from http://github.com/nuget/nugetgallery
We do want to make internal deployments of the NuGet Gallery simpler over time. Couple that with the extensibility point we're discussing here where the registry can be used to augment users' nuget.config settings, and the enterprise space should improve quite a bit.
Mar 25, 2013 at 3:37 AM
Of course -- but my point was that when designing this API, please keep in mind that private repositories are a critical part of the ecosystem and should be taken into consideration. The nuget.exe client shouldn't require, for example, that server implement "personally curated feeds" as these make little-to-no sense in enterprise environments. I'll post specifics to the GH issues.

Also, since you're a NuGet/Microsoft representative... I have a small request. There are currently four commercial private repositories (Artifactory, MyGet, Nexus, ProGet), and I obviously represent one. When discussing things like options for private feeds, could you use wording like "commercial repositories", "a third-party repository", or something instead of saying "to solve this, use vendorx" (like your above response)?

I give a lot of talks on NuGet, and while it's obviously in my best interest that people only use ProGet, I always mention our peers -- we compete, but we all take a different approach, and I think the audience should be given a chance to judge for themselves.