Stash style update for Github repos

[Phuket2]

I think this question is aimed at the StaSh gurus/authors here.

My question comes about with this senerio

I have just copied a snippets tool from a repo

I would like to stay up to date with features/fixes etc...get the current version....
I am assuming just cloning a repo is not the way to go. Maybe if one big project. But typically users here have many tools in the same repo.
But let's say the repo owner made a manifest file for named entities inside the repo. Eg. There was a file called 'snippets.manifest' in the repo. Let's say it's a json file that contains version info, a list of files inside the repo that represents the 'package'. Maybe other information about API compatibility... I am sure you get my meaning.

Then what would be cool in stash or using the stash objects, you could say pass the repo url as well as the base name of the 'package' and a cmd like 'version' or 'update' or 'install' etc...

In my mind, this is a huge thing missing for a language like Python.Yes you have pip install for large frameworks/libs. But nothing for the tiny/small/... style offerings.

Look, maybe I have it all wrong. Would not be the first time. Maybe it's all in git. I am not sure. But my feeling git is project centric. Not really that helpful for many small codebases contained in the same repo.

Anyway, it's just an idea.

[Webmaster4o]

If you made it with git clone, you can do this with git pull origin master. Is this what you're looking for?

[Phuket2]

@Webmaster4o , I am not sure. But I don't think so. I there are 10 projects in the repo and I am only interested in the files that pertain to one project, I will get the whole repo right?

[Webmaster4o]

@Phuket2 You might be able to git pull specific files, I would google it.

[Phuket2]

@Webmaster4o , I will. But I still don't see it as being managed. If the snippets repo got split into several files or extra files were added to extend functionality, you would have no way to know about their existence
Edit, not repo got split. But the file

[shaun-h]

@Phuket2 first of all the repo you are talking about I will probably split that file in to others in the very near future it is starting to get a bit big.

Now for your comment, how I use git is a single repo for a each project I create otherwise how would you know within the repo which files would you need to make a tool work of you had lots of tools in a single repo.

I am not sure if git has the ability to pull down individual files from a repo, but if someone knows how to do this without pulling the whole repo down first I would be interested.

[Phuket2]

@shaun-h , that's why I mention a manifest file. Possibly .json format. I understand, It can't work without something like that.

[JonB]

this is called sparse checkout, and is not currently supported in stash or dulwich, or even in github i think.

http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/

[Phuket2]

@JonB , thanks. I had a quick look at the article. Buts it's not what I am talking about.
My idea equates more to PyPI. The ability to install a lib from a github repo. Today it maybe one file tomorrow maybe 2 or more files. If a type of manifest was kept uptodate by the author it seems possible to me.
My meaning being, whilst Github seems to be for managed code, meaning the consumes are interested in the code itself. I am looking another way. The ability to stage installs from Github. The consumer has no interest in the source code per say, just want the package as you do for something like SQLite.
Maybe there is a system out there that is to publish a 10 line class or function to something like PyPI so you can just do a pip install. But I am pretty sure there is not. All I am saying there should be. Well at least in mobile programming it starts to make more sense.

Look, it's ok. I am sure things will change. So much we are using today has not caught up to how things work today. Seriously, if I was on the board of GitHub, I would be saying hey guys what are we doing to facilitate mobile platforms ? Were is the iOS GitHub app? Yeah?? Yes, today still many restrictions on developing apps on mobile devices for mobile devices. But in all honestly , how long to you think that will last. Ok, I know I go on and on....but at the exponential rate things are going, it's anyone's guess we're will be in 1,2,3,4 years from now. But its still going like moores law

[JonB]

oh i gotcha. Several stash packages auto install dependencies, using similar code... I think we could probably consolidate that code into a command to do something similar from the commandline. Also, someone mentioned it would be good to check whether user is on cellular before downloading.

[ywangd]

@Phuket2 In theory, a single GitHub repo is meant for just one project. So git clone ... and git pull afterwards are the recommended way to stay sync with the original source.

I do agree that the code sharing practice in Pythonista community is different from the standard repo usage, i.e. a single repo can host scripts from multiple projects. Now you want install only just a few scripts from a single repo.

If I understand your requirements correctly, to my knowledge, the Pythonista-Tools and its user frontend pythonista-tools-installer are currently the closet things. Script authors can register their scripts on Pythonista-Tools and the Installer reads the registration and provides one-click installations for end-users. In this case, the registration files (i.e. the markdown files hosted by Pythonista-Tools) are similar to the manifest files you described.

For an example, the MarkdownView is regitered as the follows:

...
| [MarkdownView]     | TextView with markdown editing and HTML viewing capabilities |
...
[MarkdownView]: https://github.com/mikaelho/pythonista-markdownview
...

Now an user of the Installer do not see above registration at all, s/he is just presented an UI that contains a list of available scripts and buttons to click and install them (all the parsing of registry files are done in background without user intervention).

The above example is for the entire repo, similar approach can be used for a single file (e.g. https://raw.githubusercontent.com/cclauss/Ten-lines-or-less/master/pythonista_version.py). Although the current Installer only supports single file hosted on Gist, it is not difficult to extend it to support Raw URLs of GitHub . Other things that are currently lacking are update and version check. Since GitHub does not have a system for release management, your best bet is to rely on the modification timestamp. It is not optimal but in most cases is an OK alternative.

The latest version of StaSh for Pythonista 3 has the Installer bundled as part of its installation. You can simply run ptinstaller from the terminal to launch the installer. Alternatively you can install the Installer with following one-liner:

import requests as r; o=open('ptinstaller.py','w'); o.write(r.get('http://j.mp/pt-i').text); o.close()

[JonB]

I was thinking it might be worth creating essentially a set of gh commands, which treats a repo /branch like a folder. Maybe

i.e

gh ls ywangd/stash:dev
gh copy ywangd/stash:master/*.py .

(branch would be optional, defaulting to master, and could be a tag, or commit sha). This would make it easy to, for example, copy out an image from someone's repo without getting the whole repo, or download a single script.

Or to be more general... i wonder if it makes sense to create a mount command? This would take some abstraction of the filesystem to achieve. But users could create "device drivers" which implement file system like operations (chdir, listdir, copyfile, movefile, open, read, write, etc) then after mounting a device (say, a github repo, or a dropbox folder), other commands should be able to access the files transparently.

[ywangd]

@JonB Adding more sub-commands to gh seems to be a great idea and could be very useful! One minor note is that the *.py won't work as you expected as the shell will expand the wildcard before passing it to the command. You'll need to quote them to avoid wildcard expansion like gh ywangd/stash ls '*.py'.

The mount idea sounds fantastic but I suspect a shell-wide mount is possible. It seems impossible for an arbitrary script to work with the mount. A script has to explicitly written to support the mount device. The shell has no way to make the mount look like an inherent part of the file system, i.e. if a script just use os.listdir(), it won't be aware of the mounted device.

My suggestion is that mount can also be a sub-command of gh. So after running gh mount ywangd/stash, other sub-commands will implicitly work with the mounted repo, i.e. you can now just run gh ls and skip the owner/repo part.

Things are relatively easier if you only want list (ls), download and upload (cp) files. The update is more involved as it requires a set of metadata to be stored along with each downloaded file.

[Phuket2]

@ywangd , thanks for the great reply. I guess I should be a little clearer. My requirement Persae is not so important unless it's in line with the community's requirements. The only thing I would mention about your reply above, is I thought the registration file you illustrate, I thought could maybe a json file or at least more structured template. As you point out versions etc, not easy to track this way. But if this file the developers publishes with contained some richer information, it would allow the tools to be smarter. Could have a Pythonista ui to help manage the file for simplicity and consistency checks.

[Phuket2]

@ywangd , just looked at the ptinstaller , that's nice

Edit: it's a huge ease of use already with that command.