NuGet 1.6 Gallery Endpoint Changes

Coordinator
Nov 15, 2011 at 11:36 PM

Hi All,

As you know, we've been working on a new implementation of the gallery that supports the changes coming in NuGet 1.6. For example, we will support SemVer for packages. There's an upcoming change to SemVer that's not yet released, but will be soon. We based our implementation on that spec.

More importantly, we're simplifying the URL structure for NuGet 1.6 endpoints for publishing packages. These are the endpoints that nuget.exe and package explorer will point to in order to publish packages. We'd also like to do the same for the OData feed, but we're going to save that for 1.7 as we're trying to avoid scope creep.

Here's the spec, please check it out: http://nuget.codeplex.com/wikipage?title=NuGet%201.6%20Gallery%20Service%20Endpoints

Thanks!

Phil

 


Developer
Nov 16, 2011 at 7:12 AM

Hi Phil, 

Quick questions that I don’t see answered explicitly:

-       Does this count for the consumption of packages as well? E.g. when I add http://www.myget.org/F/somefeed to VS2010, will it consume that feed or will it try to consume http://www.myget.org/F/somefeed/api/v2 ?

-       If I push to http://www.myget.org/F/somefeed, will NuGet use that as the endpoint for push? Or will it be http://www.myget.org/F/somefeed/api/v2/package (which I would prefer)

In short: if I have this URL to a feed, what will all the required endpoints be? http://www.myget.org/F/somefeed 

Best regards,
Maarten

Nov 16, 2011 at 8:34 AM
Edited Nov 16, 2011 at 8:34 AM

Hi guys,

My understanding of the spec is that for everything other than one-feed-per-domain, we'll need to specify the full endpoint path, because nothing will be appended automatically? That's going to degrade the experience of MyGet and SymbolSource users a bit...

One idea that comes to mind is to do redirects on the server-side based on User-Agents:

  • NuGet Add Package Dialog/1.5.20902.9026 (Microsoft Windows NT 6.1.7601 Service Pack 1)
  • NuGet Command Line/1.5.20830.9001 (Microsoft Windows NT 6.1.7601 Service Pack 1)
  • NuGet Package Explorer/2.5.0.0 (Microsoft Windows NT 6.1.7601 Service Pack 1)

But that doesn't give us a way to differentiate between package publishing and feed consumption.

Cheers,
Marcin 

Developer
Nov 16, 2011 at 11:45 AM
Edited Nov 16, 2011 at 12:09 PM

I would vote to do the following:

In short: the supplied URL is the base to which /api/v2/whatever is added, unless it is specified explicitly. This should allow full support for MyGet and SymbolSource.

Next to this, I noticed the API key will be shipped as a HTTP header instead. Is that correct?

Nov 16, 2011 at 12:19 PM

Maarten, I think the original goal was to allow backward compatability, i.e. to be able to use a 1.6 client to push to a 1.5 server. In your scheme the 1.6 client would try to append 1.6 paths to full 1.5 urls. To make this work it would have to know the list of all possible suffixes (current api version and all previous versions) and check those to see if it should append the default (current version, api/v2/something).

Nov 16, 2011 at 12:29 PM
This seems pretty sensible to me, and means NuGet clients like myself don't need to do too much guessing.

Paul

From: maartenba
Sent: 16/11/2011 11:45
To: Paul Stovell
Subject: Re: NuGet 1.6 Gallery Endpoint Changes [nuget:279575]

From: maartenba

I would vote to do the following:

In short: the supplied URL is the base to which /api/v2/whatever is added, unless it is specified explicitly. This should allow full support for MyGet and SymbolSource.

Developer
Nov 16, 2011 at 12:29 PM

That would still be possible: if my feed endpoint remains "http://www.myget.org/F/somefeed", I can keep the routes for the old 1.5 urls as well. No breaking changes and support for both 1.5 and 1.6 clients, even without explicitly specifying the publish URL.

Nov 16, 2011 at 12:37 PM
maartenba wrote:

That would still be possible: if my feed endpoint remains "http://www.myget.org/F/somefeed", I can keep the routes for the old 1.5 urls as well. No breaking changes and support for both 1.5 and 1.6 clients, even without explicitly specifying the publish URL.

Maintaining old routes only takes care of client=1.5 and server=1.6, I was talking about the opposite.

Consider a client=1.6 and server=1.5 and pushing to the URLs:

The section "Workaround for NuGet 1.5 galleries w/ NuGet 1.6 exe" of the spec talks about this scenario.

Developer
Nov 16, 2011 at 1:08 PM
Edited Nov 16, 2011 at 1:08 PM

Sidenote: This brings back my idea of NuGet feed metadata: why not have a http://www.myget.org/F/somefeed/$nugetmetadata URL which returns:

  • The feed name "Official NuGet package source" or "Maarten's feed"
  • The feed version (v1, v2, ...) so the client knows how to work with it
  • The URL to related feeds, for example feeds where packages can find dependencies, or the SymbolSource URL, or ...

Base on those, it's easy to determine the correct URLs. But I digress :)

Nov 16, 2011 at 1:21 PM

I'm afraid this digression would be the only clean solution. Well, now we need to wait for the US to wake up and respond ;)

Coordinator
Nov 16, 2011 at 6:17 PM

Note that the NuGet 1.6 server won't put the API Key in the URL any longer. So while you do have to specify the full URL in your case, it's not as long as it used to be.

The problem with always appending /api/v2 is for folks hosting internal NuGet galleries. Using NuGet 1.6 to post to them would be impossible without them upgrading them. While we want folks to upgrade their servers, that's a bigger undertaking than upgrading nuget.exe and we don't want to force them to do it on our timeline necessarily.

Coordinator
Nov 16, 2011 at 6:18 PM

For MyGet, I wonder if a NuGet.exe plugin could make that a better experience so folks don't need the full URL.

Developer
Nov 16, 2011 at 7:17 PM

Back to the original question :-)

 

- Does this count for the consumption of packages as well? E.g. when I add http://www.myget.org/F/somefeed to VS2010, will it consume that feed or will it try to consume http://www.myget.org/F/somefeed/api/v2 ?

- If I push to http://www.myget.org/F/somefeed, will NuGet use that as the endpoint for push? Or will it be http://www.myget.org/F/somefeed/api/v2/package (which I would prefer)

In short: if I have this URL to a feed, what will all the required endpoints be? http://www.myget.org/F/somefeed

 

I'm a bit affraid that users will need "yet another" URL as MyGet (and SYmbolSource) use part of the path to identify the feed. The current spec linked to in the opening post of this topic really limits flexibility in this as it only allows one NuGet feed per domain?

Coordinator
Nov 16, 2011 at 7:19 PM

Unfortunately, no. We will look at doing that for NuGet 1.7.

Coordinator
Nov 16, 2011 at 7:20 PM

I should clarify. The URL to consume feeds is not changing in the same way that the URL to push feeds are. We're trying hard to wrap this release up and can look at doing something like that for 1.7.

Coordinator
Nov 16, 2011 at 7:22 PM

If I push to http://www.myget.org/F/somefeed, will NuGet use that as the endpoint for push? Or will it behttp://www.myget.org/F/somefeed/api/v2/package (which I would prefer)

No, because the URL has a path portion, it'll push to exactly: http://www.myget.org/F/somefeed

Examples:

nuget push http://www.myget.org/F/somefeed

Pushes to http://www.myget.org/F/somefeed

nuget push http://www.myget.org/

Pushes to http://www.myget.org/api/v2

Make sense? Remember, this logic is in the NuGet.exe client.

Developer
Nov 16, 2011 at 7:27 PM

So if I want to make it easy for users I would just internally redirect POST and DELETE to http://www.myget.org/F/somefeed towards http://www.myget.org/F/somefeed/api/v2/package ? Would make sense to do.

Nov 16, 2011 at 7:29 PM

May I ask why are you guys pushing this now in 1.6? Why not postpone it altogether to 1.7, to find a good solution for both the push URL and feed consumption URL? Aren't you afraid that the scheme will need to change again in 1.7 and that we'll all end up with 3 schemes out there instead of 2?

Nov 16, 2011 at 7:32 PM
Edited Nov 16, 2011 at 7:32 PM

What will the exact request sequence be? At this moment nuget.exe push -Source http://server/path first makes a GET to /path (I have no idea what for, because we return 404 at SymbolSource and it still works) and then a POST to /path/key/whatever. Will it now be only POST? 

What about nuget delete? What URLs will that produce?

Coordinator
Nov 16, 2011 at 7:32 PM

@maartenba sure. That could work. Since you have a custom implementation of the gallery, you could just route http://www.myget.org/F/somefeed directly to the controller actions that handle those requests.

@tripleemcoder because if we wait till 1.7, as soon as 1.6 comes out, anyone who upgrades nuget.exe is broken with no workaround. We had to version the feed because of the pre-release SemVer support made it impossible to have the feed URL stay the same and be backwards compatible.

For consuming the feed, we have a way to change the URL that will be backwards compatible. We can use a discovery mechanism much like the way RSS readers discover the RSS feed endpoint. Handling the read case is much easier than the push/delete case.

Coordinator
Nov 16, 2011 at 7:33 PM

@TripleEmcoder not sure why nuget push makes a GET post. As for the delete case, it'll be an HTTP DELETE request to the same URL. 

Nov 16, 2011 at 7:35 PM
haacked wrote:

@TripleEmcoder not sure why nuget push makes a GET post. As for the delete case, it'll be an HTTP DELETE request to the same URL. 

I should have asked more specifically: how will the package name and version for deletion be passed?

Nov 16, 2011 at 7:39 PM
haacked wrote:

@tripleemcoder because if we wait till 1.7, as soon as 1.6 comes out, anyone who upgrades nuget.exe is broken with no workaround. We had to version the feed because of the pre-release SemVer support made it impossible to have the feed URL stay the same and be backwards compatible.

Can you elaborate on this? I don't see why SemVer would break the push endpoint (in terms of URLs, in terms of backend processing and logic - that's obvious), which probably means that I don't understand something. And I really should for SymbolSource's sake ;)

Developer
Nov 16, 2011 at 7:48 PM
Edited Nov 16, 2011 at 8:10 PM

*Edit: Corrected some obviously wrong responses.

 

To clarify your question:

  • Does this count for the consumption of packages as well? E.g. when I add http://www.myget.org/F/somefeed to VS2010, will it consume that feed or will it try to consume http://www.myget.org/F/somefeed/api/v2 ?
    We will consume http://www.myget.org/F/somefeed
  • If I push to http://www.myget.org/F/somefeed, will NuGet use that as the endpoint for push? Or will it be http://www.myget.org/F/somefeed/api/v2/package (which I would prefer)
    It would use the former without appending anything. For sake of compatibility with older clients, NuGet gallery would keep existing routes around for a while. NuGet.Server would not be doing this. Additionally, the api key would be sent as part of a header named X-NuGet-ApiKey.
  • In short: if I have this URL to a feed, what will all the required endpoints be?
    We're implementing the following endpoints in the upcoming gallery:
    • Push: POST http://nuget.org/api/v2/package
    • Delete\Unlist: DELETE http://nuget.org/api/v2/package/{id}/{version}
    • Download: GET http://nuget.org/api/v2/package/{id}/{version}
Developer
Nov 16, 2011 at 8:07 PM

Phil says: nuget push package apikey -Source http://www.myget.org/F/somerepo/ pushes to http://www.myget.org/F/somerepo/
You say:  nuget push package apikey -Source http://www.myget.org/F/somerepo/ pushes to http://www.myget.org/F/somerepo/api/v2/package

Help :-)

Developer
Nov 16, 2011 at 8:11 PM

Sorry about that. That was patently incorrect. Fixed my answer :)

Coordinator
Nov 16, 2011 at 9:21 PM

Sorry for all the confusion. I updated the original spec with more details. Thanks for participating in the discussion. It's a big help!

Nov 16, 2011 at 9:44 PM

A new question then: will it be possible to hard delete a package through the gallery? I understand the intentions, but I also remember the same issue at SymbolSource. People simply want to be able to delete stuff.

Coordinator
Nov 16, 2011 at 9:53 PM

We won't support that in 1.6. Possibly in a later version.

Right now, we're taking the approach that Ruby Gems does: http://help.rubygems.org/kb/gemcutter/removing-a-published-rubygem

Nov 16, 2011 at 10:01 PM
Edited Nov 16, 2011 at 10:02 PM

Ok. Like I said, I agree with this logic, and tried to express it numerously before, here for one: http://www.symbolsource.org/Public/Blog/View/27. But then we ended up implementing it nevertheless, as demand was really high.

So expect an uprising ;)

Developer
Nov 17, 2011 at 6:32 AM
Edited Nov 17, 2011 at 6:34 AM

Thanks for clarifying, although I now expect a fool proof URL scheme in 1.7 ;-) (eg support path segments to identify a feed as well, the domain level may be too greedy)

MyGet has been updated and should, with the touch of a button, switch over to 1.6 whener you say it's ready.

Nov 17, 2011 at 1:10 PM
Edited Nov 17, 2011 at 1:11 PM

SymbolSource is almost ready too, but we're low on resources at the moment and will need extra 1-2 days for testing and final corrections.

Also, nuget.exe 1.5 handled server errors by displaying anything sent back from the server with a 500 status code. The current build of 1.6 ignores all content. We would very much like to be able to send back multi-line descriptive errors like we have so far. This greatly helps in diagnosing symbol package validation errors.

Coordinator
Nov 17, 2011 at 5:33 PM

We're not ready to go this week either. Thanks for being responsive to these changes! :)

Developer
Nov 17, 2011 at 6:36 PM

Aha, so there's time for the endpoints and features mentioned in this thread ;-)

Coordinator
Nov 17, 2011 at 6:37 PM

No, we need that time to implement and test what we just agreed upon! :P

Nov 17, 2011 at 6:54 PM

"Agreed upon" might be a slight overstatement :P

Coordinator
Nov 17, 2011 at 7:20 PM

Ok, I have a bit of a concern. The NuGet Gallery project restores one package from a private MyGet feed. We just updated the gallery to use NuGet.exe 1.6 to restore that package. Well of course that fails, because MyGet hasn't yet pushed their 1.6 changes live.

Now, as we all discussed, there's a workaround. I can put my API Key in the URL and craft it to use the full URL temporarily. But when MyGet switches to 1.6, my build breaks again. How annoying!

Could I convince you MyGet folks to consider a v2 versioning scheme for your URLs as well? :) That way my existing build using NuGet.exe 1.5 doesn't break. And when I upgrade to NuGet.exe 1.6, I can simply point it to the v2 feed. That would allow you to start publishing the 1.6 compatible feed URLs today. :)

I believe this is the class of computer science problems known as the "chicken / egg" category. :P

Developer
Nov 18, 2011 at 11:57 AM

So wait, NuGet uses MyGet? Awesome :-)

Regarding your question: the v1.6 compatible feed is there already: http://www.myget.org/F/chucknorris/api/v2. Pushing is also supported according to the above schema and the fact that the API key comes in a header. In that case, http://www.myget.org/F/chucknorris/api/v2/package + a POST are your friend.

More awesomeness: does this mean MyGet is currently in the lead for the definition of NuGet protocol endpoints? ;-)

Coordinator
Nov 18, 2011 at 5:11 PM

Ha! Yes, you are in the lead!

And yes, NuGet Gallery uses MyGet to host the latest build of NuGet.Core we use during development of the NuGet Gallery. Inception!

It’s all so recursive. ;)

Developer
Nov 18, 2011 at 5:20 PM

I thought inception was about a VM in a VM but apparently inception is about a NuGet package in a MyGet feed built using a NuGet package. Mind. Blown.