FTP sync

Gerzer

@ccc I replied to your comments on the Gist but I don't think Gist comments send out emails, so just letting you know :). Not to pressure you into responding, though.

ccc

Yet another reason to use repos instead of gists ;-). I responded with a clarification. My preference is json for machine written files and yaml for human written config files.

Gerzer

YAML it is!

Webmaster4o

When trying to use this script, I get the same output repeated over and over. I left it running for 5 minutes and saw no change.

The output that is repeated is

*cmd* 'CWD /.'
*resp* '250 OK. Current directory is /'
*cmd* 'CDUP'
*resp* '250 OK. Current directory is /'
*cmd* 'CWD /.'
*resp* '250 OK. Current directory is /'
*cmd* 'PWD'
*resp* '257 "/" is your current location'
*cmd* 'TYPE A'
*resp* '200 TYPE is now ASCII'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (74,81,82,170,138,41)'
*cmd* 'NLST /'
*resp* '150 Accepted data connection'
*resp* '226-Options: -a \n226 5 matches total'
*cmd* 'PWD'
*resp* '257 "/" is your current location'

This exact text repeats over and over, except for the fact that the first time lists *cmd* 'CWD /' instead of *cmd* 'CWD /.'

Is it working properly and merely taking a while, or is there something wrong?

My settings are:

Using ftptest.net, my test succeeds with the warning Warning: The entered address does not resolve to an IPv6 address. It saysp

Your server is working and assorted routers/firewalls have been correctly configured for explicit FTP over TLS as performed by this test. However there have been warnings about compatibility issues,
not all users will be able to use your server.

Not sure if this is the source of the problem.

I own this server, it was configured through cpanel on my domain, deentaylor.com

Gerzer

It just takes awhile sometimes. :) I'm working on optimizing the custom FTP server directory walking mechanism, it is currently somewhat inefficient (many FTP servers don't implement a "walk"-type command even though one is defined in some versions of the protocol).

Webmaster4o

My only other question is, how do I know when I'm pulling files from the server to my device and when I'm pushing files from my device to the server?

Gerzer

If the command is RETR, then the script is retrieving a file from the server. STOR means the script is storing a file on the server. Also, I was incorrect about the existence of a "walk"-type command in the FTP protocol, so I updated my previous post.

Webmaster4o

Ok. I'm working on a github-based alternative, because that will have built-in functionality to view older versions of code. I'm thinking something like a multi-device version of Time Machine on a Mac.

Webmaster4o

@Gerzer My idea for an improvement to FTP sync is that you could compress all files first (as in @omz's pythonista-backup.py) before uploading to the FTP server. This would prevent having to upload files one by one, and also reduce file size on the server. Then the script could simply retrieve and decompress files on the other end.

Gerzer

That sounds interesting. How much value is there for editing on the FTP server directly? If that is an important feature, then I could create a desktop app to access the files. Perhaps that's not necessary, however. What do you think?

ccc

It might be cool @Webmaster4o to also have a standalone script that walks the Pythonista directory structure creating a tarball (.gz, .zip, whatever) as it goes. This single file archive could then be transferred via FTP, Webdav, Dropbox, email, etc. I know that the Pythonista zipfile module has some upper limit on filesize.

It might also be cool to have a script that shows the 10 largest files in the Pythonista directory structure.

Webmaster4o

@omz's pythonista backup script creates a zip file of the entire structure with shutil.make_archive. He stores this in a temp file (otherwise it grows in size forever and never finishes) and then moves it to the main directory. I'm actively working on an FTP sync that compresses first, I've just written a file browser with UI to browse the contents of a zip file without extracting first. The service will upload the entire zip, then allow the user to choose individually which files to extract from the archive. I'm also hoping to have a system by which multiple (5?) backups are kept on the server and you can choose from when you want to restore a file.

MartinPacker

I think the limit for a zip file is 2GB (though there might be a version that allows for more) @ccc.

Webmaster4o

Compressing my entire library yeilds about 70MB… I don't think that in compressed form 2GB is realistic

medufacaxperiae5

This post is deleted!