Controlling where packages are placed locally

Oct 7, 2010 at 12:06 PM

Loving nupack so far but i have a feature request :)

It would be very useful if you where able to control where nupack places the packages locally. Right now they end up in $SolutionDir\Packages but in our repo we have a bunch of solutions and above that, a folder called BinaryLibraries where all external stuff go. Beeing able to share the packages across solutions would reduce the number of duplicate binary files that would have to go inte source control.

Im not sure if that information should go in the packages.config file although that seems to be the place to put it. Perhaps a default value should also be in the global VS settings

Oct 7, 2010 at 12:28 PM

I agree with this as well.   I'm pretty anal about where I put my external stuff.   I always create an Externals folder in my solution directory.  From there I create a solution folder and copy everything from the externals folder into that so I can make sure that the version of the external assembly I need is source controlled along with the solution itself.

Oct 7, 2010 at 3:13 PM
Yeah this is going to be one of those areas where no one really agrees, but in the end we all want the same thing I believe. We want our dependencies in source control.

If that is what you want then know that this is how it works by default. It may not be the name you like (I use 'lib' for example) but it still achieves the same goal. :)

Also I think its important to note that stylistically we are assuming one SLN per source control repo at the moment.

-d

On Thu, Oct 7, 2010 at 6:28 AM, stylezhouse <notifications@codeplex.com> wrote:

From: stylezhouse

I agree with this as well.   I'm pretty anal about where I put my external stuff.   I always create an Externals folder in my solution directory.  From there I create a solution folder and copy everything from the externals folder into that so I can make sure that the version of the external assembly I need is source controlled along with the solution itself.

Read the full discussion online.

To add a post to this discussion, reply to this email (nupack@discussions.codeplex.com)

To start a new discussion for this project, email nupack@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Oct 7, 2010 at 3:32 PM

When I first saw this project, I wanted absolute control as well. While I would not pick Packages/ect...  At this point, I would rather have a convention and stick to it.  I have a different perspective than I had a few years ago.  If we can have a convention that works, than that just makes the entire eco system easier to recognize, for .Net developers.  I do understand where your coming from, but I would rather see efforts pushed toward higher level features versus overide configuration.

Oct 8, 2010 at 1:24 PM
Edited Oct 8, 2010 at 1:28 PM
drusellers wrote:
Yeah this is going to be one of those areas where no one really agrees, but in the end we all want the same thing I believe. We want our dependencies in source control.
If that is what you want then know that this is how it works by default. It may not be the name you like (I use 'lib' for example) but it still achieves the same goal. :)
Also I think its important to note that stylistically we are assuming one SLN per source control repo at the moment.

 ...witch is why it should be configurable :) yes, i want dependencies in source control, but i dont want duplicate files in source control.

i think the assumption that there is only one solution per repository is completely false... you honestly think a company is going to share all their projects in a single solution? surely you cant mean that :)

 

Oct 8, 2010 at 1:27 PM
Edited Oct 8, 2010 at 1:29 PM
erichexter wrote:

When I first saw this project, I wanted absolute control as well. While I would not pick Packages/ect...  At this point, I would rather have a convention and stick to it.  I have a different perspective than I had a few years ago.  If we can have a convention that works, than that just makes the entire eco system easier to recognize, for .Net developers.  I do understand where your coming from, but I would rather see efforts pushed toward higher level features versus overide configuration.

but you dont have a convention that works. not at scale at least.. and no convetion works in -every- scenario. in any case if you want to follow the convetion, you could just leave the settings at default. companies already have conventions for binary libraries and nupack should support integration with those conventions

Oct 8, 2010 at 2:51 PM
Should it be configurable. eh~ We all have our conventions and standards and I would argue (and this is only an opinion of course) that it would be better for the .net ecosystem as a whole if we all came into more of a standard solution, ala ruby and python and java. Not that we need follow their conventions, just that if we had one. It would be nice.

Duplicate files in source control, well I don't know what to say to that, disk is cheap and its just not something I worry about. So we have a different value system there. :( but its cool. :)

Where I work we maintain around ~100 in house built solutions. Each solution follows a file layout that is close to what you will see in my OSS projects as well. We have one solution (which may contain X number of cohesive applications around the solution) per repository which gives us around 100 repositories in SVN as well. This has worked well for us for the last 3 years, and we currently plan to continue down this road (of course I am always up for improving my game).

I am not sure what you mean by 'share all their projects in a single solution' so I can't speak to that directly.

Anyways, I do think it is interesting how we all do things differently to ultimately reach the same goal.

-d



On Fri, Oct 8, 2010 at 7:24 AM, aL3891 <notifications@codeplex.com> wrote:

From: aL3891

drusellers wrote:
Yeah this is going to be one of those areas where no one really agrees, but in the end we all want the same thing I believe. We want our dependencies in source control.
If that is what you want then know that this is how it works by default. It may not be the name you like (I use 'lib' for example) but it still achieves the same goal. :)
Also I think its important to note that stylistically we are assuming one SLN per source control repo at the moment.

 ...witch is why it should be configurable :) yes, i want dependencies in source control, but i dont want duplicate files in source control.

i think the assumption that there is only one solution per repository is complete false... you honestly think a company is going to share all their projects in a single solution? surely you cant mean that :)

 

Read the full discussion online.

To add a post to this discussion, reply to this email (nupack@discussions.codeplex.com)

To start a new discussion for this project, email nupack@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Oct 8, 2010 at 2:52 PM
why doesn't the convention scale? could you define scale pls?
settings at default?

-d

On Fri, Oct 8, 2010 at 7:27 AM, aL3891 <notifications@codeplex.com> wrote:

From: aL3891

erichexter wrote:

When I first saw this project, I wanted absolute control as well. While I would not pick Packages/ect...  At this point, I would rather have a convention and stick to it.  I have a different perspective than I had a few years ago.  If we can have a convention that works, than that just makes the entire eco system easier to recognize, for .Net developers.  I do understand where your coming from, but I would rather see efforts pushed toward higher level features versus overide configuration.

but you dont have a convetion that works. not at scale at least.. in any case if you want to follow the convetion, you could just leave the settings at default. companies already have conventions for binary libraries and nupack should support integration with those conventions

Read the full discussion online.

To add a post to this discussion, reply to this email (nupack@discussions.codeplex.com)

To start a new discussion for this project, email nupack@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com


Oct 8, 2010 at 3:23 PM

but you dont have a convetion that works. not at scale at least.. in any case if you want to follow the convetion, you could just leave the settings at default. companies already have conventions for binary libraries and nupack should support integration with those conventions

There is a lot of context about why this does not scale for you, that you are leaving out.  I do believe that this may not work for you.  I work at a very large company with a lot of developers, on multiple continents.  I have also, in the past worked at small shops on small teams, as well.  There is a cost to duplication of dependencies in source control, but there are also tradeoffs that come with forcing a shared dependency model in your source control.  Lets talk through some specifics and see what the heart of the problem is.
 
There are a few other ways to tackle this problem including build targets and local system caches which could reduce the weight on a source control system. I think those options need to be thought through before jumping to a conifurable folder name for storing packages.  Configuration, when it is not checked in with the solution leads to solutions that do not build on a clean machine after getting the codz from source control.  That should be avoided at all costs, and that is the scenerio I want to help prevent.  What are you really trying to enable?
Oct 8, 2010 at 6:34 PM
Edited Oct 8, 2010 at 8:51 PM
drusellers wrote:
Should it be configurable. eh~ We all have our conventions and standards and I would argue (and this is only an opinion of course) that it would be better for the .net ecosystem as a whole if we all came into more of a standard solution, ala ruby and python and java. Not that we need follow their conventions, just that if we had one. It would be nice.

Duplicate files in source control, well I don't know what to say to that, disk is cheap and its just not something I worry about. So we have a different value system there. :( but its cool. :)

Where I work we maintain around ~100 in house built solutions. Each solution follows a file layout that is close to what you will see in my OSS projects as well. We have one solution (which may contain X number of cohesive applications around the solution) per repository which gives us around 100 repositories in SVN as well. This has worked well for us for the last 3 years, and we currently plan to continue down this road (of course I am always up for improving my game).

I am not sure what you mean by 'share all their projects in a single solution' so I can't speak to that directly.

Anyways, I do think it is interesting how we all do things differently to ultimately reach the same goal.

Yes, we all have our convetions, so who is to say the convetions of the nupack team is the right ones? a system should conform to its users , not the other way around. by the same token you can argue that people should all use the same code standard and that would be "simpler" or "better" but people have different preferenses and nupack should not force the preferences of its developers onto its users..

You do that att your workplace? great. But im pretty sure there are a whole lot of people out there that -dont- create a new repo for every solution (when you branch you dont create another repo for example, do you?) just because it works for you doesnt mean it works for every developer in the [.net] world

yes you -can- have duplicate files in source control, its not disk space im worried about [network speed is a far bigger problem]. but it clutters up the repo and to me its just plain ugly.

@why it doesnt scale

it doesnt scale since you're assuming there is only one solution in the repro, i have a hard time anything but tiny hobby repos that doesnt have more that one solution, if not only for branching

Oct 8, 2010 at 6:43 PM
Edited Oct 8, 2010 at 8:47 PM
erichexter wrote:
There is a lot of context about why this does not scale for you, that you are leaving out.  I do believe that this may not work for you.  I work at a very large company with a lot of developers, on multiple continents.  I have also, in the past worked at small shops on small teams, as well.  There is a cost to duplication of dependencies in source control, but there are also tradeoffs that come with forcing a shared dependency model in your source control.  Lets talk through some specifics and see what the heart of the problem is.
 
There are a few other ways to tackle this problem including build targets and local system caches which could reduce the weight on a source control system. I think those options need to be thought through before jumping to a conifurable folder name for storing packages.  Configuration, when it is not checked in with the solution leads to solutions that do not build on a clean machine after getting the codz from source control.  That should be avoided at all costs, and that is the scenerio I want to help prevent.  What are you really trying to enable?

yes it works but its ugly. we have partners overseas with an increeedibly slow internet connection and checking out binary dependencies are already a pain. having a bunch of duplicate files would make that even worse. Have the nupack team done research into every .net shop and their conventions and concluded that your system is the best? if not, what gives you the right to dismiss any other approach other than yours as "confusing". There is no guarantee that people will not brake the build with nupacks current approach either, nothing is forcing you to check in the current config files or binaries. 

Nupack is great but people are not going to restructure their entire repos just to conform to what you prefer.. what am i trying to enable? user choice, thats what.

-edit-

looks like i misread your post slightly, you didnt infact call diffrent folder structures than nupacks, confusing. my bad. but i belive my point is still valid, no matter what apporach is used, people will still break the build and i belive the biggest risk of that happening is forcing people to work in a way they are not used to..

Oct 9, 2010 at 1:24 PM

i created WI#215 for this suggestion, perhaps we can track it there instead :)

Oct 10, 2010 at 12:29 AM

I have also been thinking along these lines - but for different reasons.

I think configuration is needed, failure to do so will be a hindrance to adoption. 

A lot of corporations have policies and guidelines on project structures - a simple configuration i.e. where do I put the packages folder and what do I call it, is far easier than trying to change these policies.

We have a project config file in the solution - surely this can be configured to hold information of where the packages should be downloaded to (or found if already downloaded) and (another option) which local/remote package repositories should be used.

 

 

 

 

 

Oct 10, 2010 at 12:53 AM

I agree that this in an area worth looking at at some point, but I think it's something we'll do post v1.  We really need to keep v1 simple and focused on a core set of scenarios before spreading is various interesting directions.  I definitely would like to support a machine wide (or per user) cache of packages, and this would be part of the same discussion.

Coordinator
Oct 10, 2010 at 1:12 AM
I'll start a "spec" page in the wiki and we can work out the design for this such as where are these settings stored. For example, how would NuPack know to look in a folder other than "packages".
>
Oct 10, 2010 at 2:46 PM

i think having configuration is very important. as @davidebb pointed out it might be best for future versions than the v1.

anyways here is one solution that i came up with.

it is the way how git solves configuration. (im finding it funny how all my threads points to git, but after all we can learn a lot from it as was made by Linus Torvalds who has had very much experience in managing the huge open source software - the linux kernel)

git uses .gitconfig as git configuration.

git solves it by having a user configuration at the home directory, which applies to all repositories. in windows that would be %userprofile% directory.

then each git repositories can override the default configuration by having its own .gitconfig file in the repository.

so what nupack could do is:

have default destination specified in nupack configuration which can be retrieved from the global nupack configuration (%userprofile%\nupack.config)

and if any devs wants to override that configuration, then just create another nupack.config file where the solution file exisits. and change the destination to be libs\ or what ever he/she wants.

even things like NuPack package sources url could be added there in the configuration. this allows the package sources to be added in source control. which means that i dont have to add the package source in VS manually. and it would auto change depending on the projects/solutions i use. incase i want to have that package source visible to all projects i would add it to my userprofile configuration.

and when u are workin on university computers or some other computers which have limited access, you can't overwrite certains files (files in program files dir). so having a local configuartion very much solves the problem.

so to solve the problem have following configuartions:

1. global configuration (c:\Program Files\NuPack\nupack.config)
2. user level configuration (%userprofile%\nupack.config)
3. local solution level configuration. (nupack.config where .sln file exists)

global>user level>local
each of these inherits the configuration, and the lover level can overwrite the configurartion if it wants.

local level configuration is very important too.

lets say one of my projects stores packages in "packages" folder. but i might be workin on some other open source software where it stores on "libs" folder. having local configuration would allow those specific projects to overwrite to whatever fits best.

and lets assume in office they require to store it in "resources" folder. so instead of creating a nupack.config file everytime i start a new project. i can rather set the defualt to "resources" in user profile directory in my office computer. (first time the user download the pacakge, might be nupack can auto create the nupack.config where sln file is copying the nupack.config from userprofile, as we can't assume the user profile config will be same as home compuer or other users config)

and the reason for global config is, we dont want devs to start creating nupack.config file every time they create a new project. so the best way is the get the default configuration which can be specified in the nupack.config file in the path where nupack is installed (program files?)

I think having those 3 configuration files would solve lot of configuration problems.

Oct 10, 2010 at 2:58 PM

and no using "registry" for global configuration please. that would be difficult in porting to linux/mono. or the configurations might be stored differently in linux and windows.

1. global config: installed directory

2. user config: user's home directory or user profiles directory

3. local config: solution directory

Oct 10, 2010 at 3:04 PM

The three levels of configuration is a common pattern and might be a good one to adopt. Just need to think about multiple developers and other use cases where it might fall down. For example what happens if I pull down a solution from source control and I have a different user config then I move to another machine. Are there conflicts or confusion?

Coordinator
Oct 10, 2010 at 4:34 PM

The problem that Bil states is exactly the one I worry about. Subtext is another OSS project I work on. Suppose I change the packages directory to be /subpackages. Now Bil decides to start helping with Subtext (Thanks man!) and gets latest from our repository. How's his NuPack client going to know to look in /subpackages? The only way that would work is if we put something in the root of the solution. Something like nupack.config. If this config is completely optional (as it should be) this might not be so bad. Most users could ignore it.

Putting something in the user or global config could be problematic for someone who works on multiple projects. Since most projects wouldn't use the optional nupack.config, NuPack would have to look for /packages first, then look for user/global config settings if no /packages folder was found. That starts to complicate something that was very simple.

I'd vote, if we did this, that we only did solution level config.

Oct 10, 2010 at 4:47 PM

Yup, what Phil says is the issue with the three levels of config. The only way it can work is if every package has a solution level config now so that I work my own way using my user config and any solution I pull down will work the way it was intended. So now instead of choice I'm always having an extra artifact to deal and I have to be aware of what machine I'm working on and what overrides/defaults exist. It seems like a good idea on the surface but when you pull back the onion it gets messy.

And yeah, I'll get to work on my Subtext patch to add Windows Live id integration right away.

Oct 10, 2010 at 5:41 PM
prabirshrestha wrote:

and lets assume in office they require to store it in "resources" folder. so instead of creating a nupack.config file everytime i start a new project. i can rather set the defualt to "resources" in user profile directory in my office computer. (first time the user download the pacakge, might be nupack can auto create the nupack.config where sln file is copying the nupack.config from userprofile, as we can't assume the user profile config will be same as home compuer or other users config

the issue you guys raised makes very much of sense if the user config is different. because of this i had mentioned earlier that nupack.config needs to be created automatically if it does not exist in the project (where solution file exists) by pulling configurations from higher level. (this needs to happen only when nupack.config is created for the first time.)

a change of thought to my first proposal of three levels of config.

at first i was thinking the config could be inherited and look up for higher levels if it does not exist, but now it seems it should be copied locally to the solution directory. so that means the global and user level configs are just meant to auto create the first nupack.config. if local nupack.config exists the global and user profile configs should be ignored.

might be only certain configurations such as package sources should be inherited. but then this would just increase the complexity. and devs would have to always refer to the manual on what is inherited and what is not. steep learning curve.

might be the best would be to never have the 3 levels of configurations but just solution level (local) configuration.

the only reason i came up with idea of user (2nd level) configuration is that most probably when i create projects i will sort of have simillar patterns, like my packages will always be stored in "libs" folder or in office they require me to save it in "resources" folder. and my office uses a list of source packages from a particular package repository and strictly disallows the use of open source libraries withouth approval (a huge problem when using GPL in commercial softwares).

in the other hand it also makes sense that one wouldn't be creating the a new project every 30mins. so it might just be easier to copy and paste the existing nupack.config and modify it according to the new needs. this would really simplify the whole concept of configurations. there is one and only one configuration which is where you solution file is. simple and precise. no more F1 :-)

Oct 10, 2010 at 5:47 PM

I think you all are over-complicating things. First of all, I don't think people are asking for different configurations for people working on the SAME project. That would be madness.

In the original Nu project, we had a solution level folder name .nu. This folder had a config.yaml file that specified the lib folder name. This was committed to version control and everyone used this setting for this project.

I think what people want is the ability to set a default "packages" folder for NEW projects. When a project is created, that configuration (nupack.config) is copied to the solution and committed. Everyone using the project would then use this same configuration. So basically the project creator would determine what the convention is for his project.

BTW: I'm using project same as solution.

If no nupack.config is present, then simply assume default packages folder.

Oct 10, 2010 at 5:54 PM

prabirshrestha, you were writing your reply the same time I was. I agree with you that we should simply copy nupack.config to the project. 

Oct 10, 2010 at 6:13 PM
Edited Oct 10, 2010 at 6:16 PM

what a coincidence.

the local nupack.config file in the solution directory would be the best.

some thoughts on sample nupack config like this

<packageDirectory>packages</packageDirectory>
<sources>
    <source alias="default">http://someurl.com/</source>
    <source alias="microsoft">http://nupack.microsoft.com/feed</source>
    <source alias="telerik">http://nupack.telerik.com/feed</source>
</sources>


so can have package directory and sources.

@kiliman yaml configs are pretty good, but just isn't popular among .net users.

linux softwares like having configs starting with . (.nu , .git, .irbc) but it just doesn't work so well on windows.

can't create file/dir in windows explorer starting with .

have to create a file/dir called nu then in dos/cmd

rename nu .nu

or incase using bash mv nu .nu

Coordinator
Oct 10, 2010 at 7:57 PM
Ah! I get it. The user/global level config is merely a convenience option that applies only when creating new projects as a means of setting up that initial solution level config. Makes sense to me. :)
>
Oct 10, 2010 at 8:56 PM

yups. its just for convenience.

it might be better to just have solution level nupack.config for now. (allows to concentrate on other important features).

once devs start creating more projects, then we might hear complains about to repetitive task for creating nupack.config. then we can look back again to either support the user level configuration or not to ease the configurations.

Oct 10, 2010 at 9:49 PM

So to summarize:

  • Short term: no configuration
  • Medium term: solution level config
  • Longer term: additional logic that knows how to copy user/global config the first time NuPack is used on a project

I'm good with that!

Oct 11, 2010 at 12:44 PM

This solution matches what intended in my first post [even though i might not have formulated it as eloquently] :)

Oct 11, 2010 at 4:01 PM

+1 for the 3-level config.

How timely! I was just wondering the same...

Oct 12, 2010 at 1:32 AM

+1 for globally overriding \Package location (either path relative to solution or absolute path).

In it's simplest form I would be happy with this as part of "Tools | Options | Package Manager" - this can be managed using VS and team settings then.

My 2cents

Oct 17, 2010 at 6:51 PM

Forgive me for this question, but I just found out about NuPack.

With what NuPack is to provide, package management-wise, why would you keep the dependencies provided via NuPack in source control? (particularly during development).

It seems to me that NuPack Server(s) should be the repository for your dependencies, and the clients should be able to pull-down specified packages on-demand when they are needed.

For example, I have specified package Widget-1.2.3 as a reference in my project.  When my colleague gets the project from source control for the first time and builds it (or before the build), the IDE client recognizes that the package has not been cached locally on his machine and pulls it down.  This would also happen the same way on the build server (with MSBuild or whatever).  Therefore the only the specification for the package is stored in source control for a project/solution.

The possibility of getting away from storing dependencies in source control is the main thing that piqued my interest in NuPack - am I way off base with the intent of NuPack?

Regards,

TEK

Oct 17, 2010 at 11:34 PM
That's exactly the opposite of my experience with dependencies. Putting them into the source control system makes sure everybody can just "pull & build" without constantly keeping things up to date with the right versions. Forcing a pull, which might require a network connection I don't have at the moment, seems less than ideal (and basically ensures that the system is tightly tied to Visual Studio, instead of being reasonably stand-alone).

One wrinkle, though, is that DVCS systems are notoriously terrible with storing dependencies, because of the burden of bringing down the entire tree. As your dependencies evolve, you're still bringing down every version of every dependency you've ever had.
Oct 18, 2010 at 3:56 AM
My interest is not really from the open source side of things, but more from an internal company usage scenario. I my case most of the packages that teams would use would be internal packages created by other teams. We all have different repositories and keeping dependencies in source control is just a PITA. Internet/network connectivity is not an issue at all in this case. I doubt that VS would always be necessary, only a robust package retrieval/dependency management system that can detect whether or not one has the correct version of a package cached locally or not. I'll continue to look at NuPack to see if it might fit our needs or at the very least serve as a foundation on which to build what I'm looking for. Cheers, TEK
Oct 18, 2010 at 4:20 AM
Edited Oct 18, 2010 at 4:37 AM

@ToddEdwinKing you might want to have a look at git submodules if you are using git. (https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial)

not sure if other source controls have similar ideas.

"Submodules allow foreign repositories to be embedded within a dedicated subdirectory of the source tree, always pointed at a particular commit."

"Submodules maintain their own identity; the submodule support just stores the submodule repository location and commit ID, so other developers who clone the superproject can easily clone all the submodules at the same revision. They are not to be confused with remotes, which are meant mainly for branches of the same project; submodules are meant fordifferent projects you would like to make part of your source tree,while the history of the two projects still stays completely independent and you cannot modify the contents of the submodule from within the main project"

Basically submodules might be the best answer for you. But then again, you will have to re-compile the subproject would increase the compilation time. So in your case, you might want to create another source control repository to store just the compiled binaries.

Coordinator
Oct 18, 2010 at 4:30 AM

@ToddEdwinKing I believe there's already a feature request for what you're talking about. It's not currently planned for v1, but is something we could consider later. It probably wouldn't be tool difficult to build the model you're talking about on top of NuPack. It makes me think we should consider some sort of plug-in model for these different types of behaviors when it comes to installing packages.

Oct 18, 2010 at 4:54 AM

plugin-in model is the best way to solve these types of problems, and even the future ones.

i had suggested a similar idea at http://nupack.codeplex.com/Thread/View.aspx?ThreadId=230265 (3rd one) on how to easily extend the nupack.exe command line tool.

later on instead of just getting the compiled binaries, nupack could even support getting source codes or help files as nupack extensions, who knows what will happen next.

nupack get-help elmah

nupack get-source elmah

nupack update elmah

and so on.

once the core nupack is extensible, it wouldn't be difficult to write extensions for it. (extensibility is the key)

My personal thoughts:

thou nupack uses power shell a lot to do tasks like List-Pacakge, i think it would be better to do it this way.

nupack list-packages

nupack sources -add http://nupack.microsoft.com/sources --alias microsoft

nupack sources --update

this would also help people using mono in mac/linux to easily use nupack coz then it doesn't depend on powershell.

for a person like me who prefers bash (though runnin in windows), i can't utilize the full potential of nupack. :-(

Oct 18, 2010 at 5:09 AM
ToddEdwinKing wrote:

The possibility of getting away from storing dependencies in source control is the main thing that piqued my interest in NuPack - am I way off base with the intent of NuPack?

 

The main goal of NuPack is not to find a way to avoid storing dependencies.  Instead, it's to make it super easy to download libraries along with their dependencies, and to then make it easy to update to newer versions as they become available.  So for the most part, it simplifies all those steps that you would normally to manually (and painfully!), but after that, the dependencies are typically stored in source control, as you'd typically do in the manual case.

That being said, as Haacked suggest, the workflow you mention has been raised, and we will look into supporting it as an alternative.  As BradWilson mentions, storing binaries in DVCS systems can grow repo sizes quite a bit, so I can see the reasoning for this alternate model.

Developer
Oct 18, 2010 at 5:11 AM

We do have a stand alone command line tool (it's pretty bare right now) that just builds packages but I think eventually you'll be able to do what you suggest. You can even contribute to it ;).  The VS experience however will remain powerhsell.

Oct 18, 2010 at 5:28 AM

would love to contribute.

actually i had already forked nupack to work with command line tool. still very experimental fork. trying out wats the best way to solve the extensions problems at https://hg01.codeplex.com/forks/prabirshrestha/nupackoptions/file/ba74ea3c4e12/NuPack/CommandLine/Options.cs#l135

actually i halted my command line fork for time being as @bismer had stated - "We had a long discussion about command line styles and using existing libraries (like NOptions) and landed on something that I have forked. I should have it in place this weekend." (http://nupack.codeplex.com/Thread/View.aspx?ThreadId=230265)

so i thought there was already someone working on it.

we might need to start a new discussion about the command line tool. so that everyone is on the same page.

Oct 18, 2010 at 11:39 AM

This thread seems to be spinning out of control from it's original intent. There's a lot of good discussion here but I see the topics as:

  • Controlling where packages are pulled down locally into the solution tree (original topic)
  • Command line tool (and options)
  • Dependencies in source control vs. package source
  • Embedded foreign repositories
  • Pulling down source code vs binaries

Ugh.

I agree with Brad. Based on experience, you really want your binary dependencies in your source tree. I know it sounds odd, but it's an age old problem. What happens if your "source" for that dependency goes away? Even if it's an internal package source, it might go away. It's hard to maintain things in a corporation when it's transparent to the useage of it. If you're tracking the usage of your internal repository that's great and just hope you keep the right version of that package there. Always.

Some SCMs have issues with binaries but just tag them as such and don't worry about them. Then anyone can pull the tree and build the system without worrying about if the right things are in the right place. Yes, you have to have the right complier/OS available but other than that it just works. For those that have low bandwidth issues this may be a problem but then how many times do you pull down the full repository? I would look at mirroring the repository or something for extreme scenarios like that.

As for pulling source code via NuPack I think that's a discussion for another day but to me it's a slippery slope. Many other package managers are going down that route and it just seems to overly complicate things. How do you know you have the right version of the source (people often forget to tag things correctly or at all). How do you know you have the right compiler options/tools for that source (in a continuous build scenario). I just think it's a whole can of worms that makes life really difficult in order to fix what problem?

Other than dependency collisions, package management is not a hard problem to solve. And for those collisions, the best you can do is just ask. No software in the world is going to be able to deduce all the permutations that make up the decision as to what version of log4net you should use.

Lets keep focused on the current problem and making things easier for developers and not over-architect a solution looking for a problem.

 

Oct 18, 2010 at 1:56 PM

@bsimer, I don't think you understand the issue. "For those that have low bandwidth issues this may be a problem but then how many times do you pull down the full repository?" With a DVCS, every time. Cloning a repository with any DVCS system means making a complete copy of the repository. This means the entire history. Every binary file, every version of every binary file, will be copied when you clone. Even with a fast connection this is problematic. Search for "large binary files {DVCS of your choosing}" and you'll see lots of issues with this, and lots of attempts to work around the problem. I understand your reasoning to want to place the binaries in source control, but in the end this is a choice you're making that may not be the correct choice for others, and the tooling shouldn't force this choice on anyone. Frankly, I see no issue with saying "to build, you must have Nupack installed" and relying on Nupack to manage the binaries. Worrying about the "source" going away should really be a non-issue, IMO, while the real world issues with maintaining binaries in version control, especially DVCSs, means I may not want to use that solution.

Oct 18, 2010 at 2:24 PM

@Haacked - Thanks - this is the information I was looking for.  It appears that NuPack could be a strong foundation for the type of artifact repository and dependency management tool we're looking to build our selves.

I apologize for bringing this thread further off-topic - the discussion around where to keep package contents and how that affects version control surprised me given my expectations for NuPack.

Thanks to the others that responded to my diversion, however I wasn't looking for advice on whether or not to store dependencies in version control, merely whether or not NuPack could (or would) be a possible solution to our problem.

Cheers,

TEK

Oct 21, 2010 at 8:28 PM
Edited Oct 21, 2010 at 8:32 PM
drusellers wrote:

Each solution follows a file layout that is close to what you will see in my OSS projects as well.

Can you please give us an example, how such standard layout looks like? Where do you put your build scripts? I am very curious to see how other developers layout their projects. Thanks!

Regarding convention vs. configuration: I totally agree that .NET needs a standard. Absence of conventions causes mess and confusion. A bit of convention would simplify communication and eliminate a lot of useless debates:-) 

Oct 21, 2010 at 8:40 PM
Example: 
  • top level project folder/
    •  /build
      • build files
    •  /src
      • solution.sln
      • /project folder
        • project files
      • /project folder 2
        • project files
    •  /lib
      • package name
        • package dlls
      • package name 2
        • package dlls
    •  /docs
Hopefully the formatting comes out.
Oct 21, 2010 at 8:44 PM

My project, Tree Surgeon, genearates this type of tree for you. You can read up on the philosophy and backgroud on creating such a structure (with all the nitty gritty details) on the project site here.

Oct 21, 2010 at 8:50 PM
Mike Roberts. I was looking for his name, but I couldn't pull it out of my head. Good influence to get started on how you should structure your application source. He has a PDF book out there somewhere that is a quick read as well. 
____
Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder
Oct 21, 2010 at 8:51 PM
Found the link: PDF is the last thing there: http://mikebroberts.com/articles/how-to-setup-a-net-development-tree/ 
____
Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder


Oct 22, 2010 at 7:13 PM

Just want to put in a vote for being able to specify the location of where the DLL's are placed.  A warning if they already exist at that location and are a different version would also be cool. 

Noel

Oct 23, 2010 at 3:59 AM

Just wanted to chime in here and say that I too am incredibly anal about where my external dependencies are placed.  I think NuPack (or whatever it shall be named) is a great idea, but I won't be using it until I can configure where the packages are downloaded to. A solution level config file is perfectly acceptable here. There's absolutely no reason not to do this.

Nov 4, 2010 at 1:39 AM

Having the ability to configure local package location is a core requirement in my eyes. The only problem that I see is that packages don't always contain binaries (e.g. jQuery). Static content can't be placed outside of your web project or else it'd be pretty hard to reference it statically. Is there a way to control where the jQuery files get dropped into your project?

Nov 15, 2010 at 10:01 PM

There is however an issue that many seem to be forgetting. Just because you put your dependencies in /lib doesn't mean that those manual dependencies have the same structure as the ones generated from packages. Even worse, you may have both local dependencies and packages managed by a package manager.

Defining where things go is one thing, but from what I understand, what people are *really* saying is "i already have my dependencies in /lib, i want my pacakges there *as well*".

Package location and local dependencies location are two completely different things, one is not to be touched by anything but a pacakge manager, the other one is your playground.

That's why we don't support changing where pacakges end up in openwrap, they're not to be managed by anyone but the package manager.