Better Quality & Regression Testing of NuGet before Release

Topics: Ecosystem, General
Feb 12 at 5:51 PM
NuGet Team,

Have you guys considered doing regression testing against the widely-used third-party NuGet tools like Inedo's ProGet, MyGet, etc., before releasing new versions of NuGet?

Reading any of Raymond Chen's blog, Microsoft is famous for not only thinking about ISVs , but actually making sure things work on new versions. There are only a handful of third-party products in the nuget ecosystem, and it wouldn't be that difficult to do given your team's resources.

Every single release of NuGet has broken tools like ProGet in one way or another due to big bugs or undocumented changes that would have been discovered in a quick test. Of course, some bugs (like #4000 have no work-around and thus leave things fundamentally broken... meaning all of ProGet's user who are unfortunate enough to not disable auto-upgrade on NuGet will have an obnoxious experience figuring out how to roll back.

I think dedicating a few more hours in your testing efforts would greatly improve the experience in the NuGet community.

Thanks,

Alex
Coordinator
Feb 18 at 3:38 AM
Alex,

I appreciate your bringing this up. As you know, NuGet’s test matrix is quite large:
  1. Virtually every VS Project type
  2. All paid SKUs of Visual Studio 2010 plus the Web Developer Express and Phone Express SKUs
  3. All SKUs of Visual Studio 2012
  4. All SKUs of Visual Studio 2013
  5. All languages supported by Visual Studio, applied both to Visual Studio and to Windows
  6. NuGet Gallery package sources; other OData-based package sources; file-based package sources
  7. And then we get to start pivoting on actual NuGet features!
I fully recognize that ecosystem partners (quite a large list) who build software based on NuGet also need to take much of the same test matrix into account, but they are then also burdened with testing new versions of NuGet. On the surface, it can seem reasonable for the NuGet team to take on responsibility of testing ecosystem products’ integration with NuGet to ensure they aren’t adversely affected—and I do think some level of this is true. However, for an ISV that is building (and sometimes selling) software based on NuGet, that software vendor owns the responsibility for testing their product with new versions of NuGet. That means staying connected to NuGet’s changes and consuming its CI builds (more info).

When NuGet does break the integration with other tools, we fully encourage those tools’ vendors to file issues against NuGet. Sometimes a change in behavior will be intentional, and sometimes not. For intentional changes in behavior, tool vendors are expected to update their products appropriately. For bugs, clearly filed issues that document expected vs. actual results will help us more quickly determine root cause, fix the issue, and create new regression tests in the area. Additionally, we would welcome pull requests that add more unit tests or functional tests to guard against unintentional changes.

Thanks again for reaching out on this. I hope this is helpful.
Jeff Handley
Feb 18 at 5:00 AM
Hi Jeff,

I appreciate the response. Couple points...

I disagree that the NuGet team should disavow compatibility testing of popular third-party products; there are really not that many products, and it would drastically improve perceived quality amongst NuGet users -- especially for organizations who've braved customizing NuGet themselves (gallery, server, etc).

Regardless, if it's the ISV's job to test NuGet, then how do you notify ISV's? There was no announcement on the blog that 2.8 had a release candidate available (did it?), nor was there even an announcement that 2.8 was out. I just happened to catch it on twitter or something, and then it was the usual "time to find out what broke in this version" shuffle.

Also, is there a test document available that shows the testing/qa done pre-release? I could contribute to that.

Alex
Coordinator
Feb 18 at 6:55 AM
Thus far we have relied on ecosystem members to stay up-to-date on our CodePlex project page where we update it with tentative release dates. We did post our 2.8 Beta builds to CodePlex two weeks before release and tweeted about it from the @nuget account. We also have an Ecosystem topic on this discussion forum, but it didn't seem to really get any traction. And I know that many folks consume our CI builds too.

We're going to look into getting a mailing list set up for announcements going forward. Thanks for the feedback on the lack of notifications being pushed out.
Developer
Mar 28 at 6:43 PM
Edited Mar 28 at 6:43 PM
Hi, Alex

We've published the NuGet.exe 2.8.1 RC today at https://nuget.codeplex.com/releases/view/118318. Would you please give it a try, in conjunction with the ProGet scenarios?

Please let us know if you hit any issues, thanks!
Apr 10 at 10:48 PM
Still broken guys :(

This of course was reported as such almost immediately after you released the RC in #4000... but I'm assuming it was too late to fix, and you had deadlines to hit and all that?

Either way, I'll go back my original point about low quality and regression testing. Here is a simple test that didn't even require ProGet to test ("does NuGet work with integrated Windows authentication"), but had you used ProGet to test it, it would have saved a lot of people a lot of hours and create a better experience...

I'll repeat my last offer... Also, is there a test document available that shows the testing/qa done pre-release? I could contribute to that.
Coordinator
Apr 10 at 11:00 PM
Thanks for the status update. After providing the latest build on 3/28, and posting both here and on the issue you linked to, we didn't hear any feedback about whether or not the issue was fixed. We had to proceed with the release on 4/2 despite not hearing back.

We will ensure Windows authentication scenarios are satisfactorily covered in our test plan. And we have been talking about the best way to publish our pre-release test plans.
Developer
Apr 14 at 10:35 PM
Hi, apapadimoulis,

In response to your comment:
Here is a simple test that didn't even require ProGet to test ("does NuGet work with integrated Windows authentication"),
Unfortunately the test for your scenario DOES need ProGet, since when we tested against NuGet.server with integrated Windows authentication, NuGet worked as expected.

Since I don't have access to ProGet, could you help us debug this problem? Once you get the nuget source code, set a breakpoint in method EnsureSuccessfulResponse() in file PackageServer.cs, on the throw new InvalidOperationException line. You can see the error message "Failed to process request. 'Found'." is generated only if the response status code >= 400. However, 'Found' should have status code 302, so it's a mystery to me how can this error message be generated. Please tell us what's the status code returned.
Apr 15 at 9:28 PM
Hi feiling,

Thanks for the update...
Since I don't have access to ProGet, could you help us debug this problem?
Happy to help of course! And of course anytime you'd want a license, we have no problem to provide.
Once you get the nuget source code, set a breakpoint in method EnsureSuccessfulResponse() in file PackageServer.cs
There's a lot of code to follow, but I'm not seeing the scenario of handling of 401 challenges on redirect. Seems the NTLM header is only passed in on the first request, but not when following redirects.
Developer
Apr 23 at 6:03 PM
Hi apapadimoulis,

I just found out why nuget didn't work for you, while I couldn't repro the problem:
  • When nuget POST a package to the server, it sets AllowWriteStreamBuffering to false to avoid out-of-memory problem when uploading big packages.
  • Unfortunately, on .NET 4.0, integrated windows authentication will fail with AllowWriteStreamBuffering set to false. The nuget.exe we released is targeting .net 4.0, which is why it doesn't work with integrated windows authentication.
  • However, our development environment is .NET 4.5, where this problem does not happen. This is why I had a hard time reproing the problem.
Now that the mystery is solved, we'll fix it soon. Sorry for the inconvenience.

Thanks,
Fei
Developer
Apr 23 at 10:55 PM
Hi apapadimoulis,

Another question, did you run nuget.exe push on a machine with only .NET 4.0? Running nuget.exe push on a machine with .NET 4.5 works correctly with integrated windows authentication. Just want to make sure what I'm trying to fix is the cause of your problem.
Developer
Apr 29 at 9:17 PM
Hi apapadimoulis,

We believe the problem is now fixed: we reverted the change that caused all those auth problems. The original change was to fix out-of-memory problem by disable buffering when sending PUT requests. This change is reverted, and we added a new option -DisableBuffering instead. You can try the new build at http://build.nuget.org/nuget.exe
Apr 30 at 5:42 PM

Thanks for the update feiling -- and good idea to add these edge cases as additional options, especially to help with backwards compatability.

Coordinator
May 7 at 10:29 PM
Our test plan overview is now published here:
http://docs.nuget.org/docs/contribute/nuget-client-test-plan

And we sent a message out to the NuGet Ecosystem Google group:
https://groups.google.com/forum/#!topic/nuget-ecosystem/L-PZFf7qQk4

If you haven't yet joined the NuGet Ecosystem Google group, please do so.
May 7 at 10:30 PM

Great news, thanks!!