We have been using a private Nuget feed in our CI environment since about September. Lately, the service has gotten much, much slower. It use to take a few minutes to install/update packages (we don't check them into source control) using the TeamCity plugin.
Now it can sometimes take 40min to update packages, but typically takes 20min on our larger projects.
Here are some stats for our environment:
Total pacakges: about 5500
Unique Packages: about 40
Developers: about 40
Hardware: Physical machine, Windows Server 2008 R2, dual quad core xenon procs (16 logical cores), 32GB RAM. Packages and server are on a SSD.
This is massively slowing down our CI. Project that use to take minutes to build, now can take 20 min. Developers fear getting latest packages because it is painfully slow. When I look at the Nuget server, it can be consuming nearly 60% of the CPU in the
heavy times (probably around 20 concurrent users - most of that is from our CI, not developers). On our larger solutions, we may have 30 projects and each of those projects may possibly be referencing 15 NuGet packages.
I have done some performance monitoring on my machine to see what is happening. An update call (i.e. nuget update <solution>) for one of our larger projects calls into GetPackages on the web server over 450 times. Each one of these calls seems to touch
all 5500 of our packages.
Here is a screenshot of the summary from profiling the Update on my desktop machine:
Top of the function list is NuGet.LocalPackageRepository.GetPackage, System.IO.Packaging.Package.Open, SyStem.IO.Directory.Exists, and System.IO.FileSystemInfo.get_LastWriteTImeUtc.
On installs, a huge amount of time is spent calculating the hash. We will probably be forking the source to make hashing optional. In our use case, the overhead with calculating the hash is not really justifiable since everything is going across our corporate
network and we are not too concerned about the integrity of that communication. Removing the hash calculation greatly sped up the install process.
I was wondering if anyone has any thoughts on performance of a private feed. I created the feed from the 1.6.1 tag in the repository. We use the command line to install/update packages and that is the latest publicly available version (1.6.21205.9031).
Is there anyway to more actively cache the file system info? Is this already implemented somehow and I just need to turn it on? Is there a way to minimize the service chatter for an update?