2

Closed

nuget.push throws OutOfMemoryException when pushing large packages.

description

The nuget.exe push command attempts to load the whole package into memory, which causes OutOfMemoryException when pushing large packages.
Closed May 23 at 5:47 PM by danliu
verified using nuget.exe 2.8.2

comments

chirayu wrote Sep 16, 2013 at 2:51 PM

Would you be able to post the stack trace that relates to this issue? I have got this stack trace and was wondering whether it relates to this issue:

Exception of type 'System.OutOfMemoryException' was thrown.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Server stack trace:
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at System.IO.Stream.CopyTo(Stream destination)
at NuGet.MultipartWebRequest.CreateMultipartRequest(WebRequest request)
at NuGet.PackageServer.<>c__DisplayClass6.<PushPackageToServer>b__5(Object sender, WebRequestEventArgs e)
at System.EventHandler1.Invoke(Object sender, TEventArgs e)
at NuGet.HttpClient.RaiseSendingRequest(WebRequest webRequest)
at NuGet.RequestHelper.GetResponse(Func
1 createRequest, Action1 prepareRequest, IProxyCache proxyCache, ICredentialCache credentialCache, ICredentialProvider credentialProvider)
at NuGet.HttpClient.GetResponse()
at NuGet.PackageServer.EnsureSuccessfulResponse(HttpClient client, Nullable
1 expectedStatusCode)
at NuGet.PackageServer.PushPackageToServer(String apiKey, Func`1 packageStreamFactory, Int32 timeout)
at NuGet.PackageServer.PushPackage(String apiKey, IPackage package, Int32 timeout)

dotnetjunky wrote Sep 17, 2013 at 2:44 AM

Fixed in changeset 746ca76fa61209f89bb825c623d2f68cfefd6a62

Sarkie wrote Sep 23, 2013 at 10:41 AM

@chirayu That looks to be the same one.

How big are your packages? And just check Task Manager to see it eating all the memory, the fix has been merged also.

danliu wrote Oct 30, 2013 at 5:14 PM

Found that pushing large packages such as >34M to nuget.server will throw an exception of "Failed to process request. 'Not Found'. The remote server returned an error: <404> Not Found..

while pushing small packages worked fine.

The large packages used for testing are the largest packages on nuger.org. e.g. openssl, mono.cecil.dll, mdf.framework and directxtk.

seanseviltwin wrote Dec 5, 2013 at 2:58 PM

Any chance we can get this in the next release?

feiling wrote Dec 10, 2013 at 4:51 PM

It turned out that fixing requires major code change. So we couldn't finish it in 2.8. I had to push it to 2.9, reluctantly.

feiling wrote Jan 6 at 11:24 PM

Fixed in changeset 83ed5502c90b71bbd29eb5ff741917f4906d6148

feiling wrote May 23 at 12:40 AM

Turns out turning off buffering will cause many other problems such as redirection won't be followed etc. We fixed most scenarios but there are still scenarios that just won't work with buffering off, e.g. NTLM on .NET 4.0 (which is a known issue in the .net framework). So we reverted this change, but added -DisableBuffering switch instead for people who truly need to push huge packages and run into this OOM issue.