Upload file to FTP using Python ftplib

[JessicaKirby]

Hello Pythoners,
I have exhausted myself searching for code that will do what I need, so I am hoping that someone here has my answer.

I need to automate a monthly upload cycle of maps to our FTP server.  Below is what I have attempted to patchwork together, but it will not work.  I am very new to python so please forgive any gross errors in this code.  Also,  the code is set up to send only one file as a test, but the end code needs to loop through a directory and transfer 29 maps, while overwriting the existing maps on the ftp server. 

import ftplib
from ftplib import FTP

File2Send = "V://GIS//Maps//County//11x17shd//2010BEAVER11x17shd.pdf"
Output_Directory = "//data2//ftp//pub//download//maps//"

try:
    ftp = FTP("XXX.XXX.XXX.XXX")
    ftp.login(username, password)
    file = open(File2Send, "rb")
    ftp.cwd(Output_Directory)
    ftp.storbinary('STOR ' + File2Send, file)
    print "STORing File now..."
    ftp.quit()
    file.close()
    print "File transfered"
else:
    print "An error occured"

[LoganPugh]

Use catch in place of else in this instance, that's going to be a syntax error.

For iterating over the files in a directory use os.listdir() -- examples: http://diveintopython.org/file_handling/os_module.html

[JoelCalhoun]

Jessica,

In our organization, those of us that need access were given permissions to the FTP server, presumably tied to our windows login.  This makes things simple.  We associated a network drive letter (F: ) with the FTP server path (e.g. \ftp_server\GIS_Maps).

If you are just updating the FTP maps with new versions of the same name you can try the following code: (Note: if the names are different then it will just add the new maps to what is existing on the ftp server.)

#Import system modules
import os, shutil

File2Send = "V:\\GIS\\Maps\\County\\11x17shd" #New maps folder
FTP_Server = "F:\\ftp_server\\GIS_Maps" #FTP server folder

#Get the list of files
f = [f for f in os.listdir(File2Send) if os.path.isfile(os.path.join(File2Send, f))]
#Loop through the list
for i in (f):
    if os.access(FTP_Server + "\\" + i, os.F_OK): #Check for existing file
        os.remove(FTP_Server + "\\" + i) #Remove existing file
    shutil.copy(File2Send + "\\" + i, FTP_Server + "\\" + i) #Copy new file

I hope this helps,


Joel

[ChrisMathers]

Sounds crazy but you have to trust me here. If you are going to use storbinary, the file cant be opened in binary mode. I know what you are going to say but I just figured this out in the last couple weeks myself and it just doesnt work.

fileList=[]
for i in os.listdir(dir):
if i.split('.')[1] == 'pdf' #if it ends in pdf
fileList.append(i) #put it in the list
ftp = FTP("XXX.XXX.XXX.XXX",username, password) #save a line and just put your U:P here.
for i in fileList:
file = open(i, "r") #open in normal read mode
ftp.cwd("//data2//ftp//pub//download//maps//")
ftp.storbinary('STOR %s' % i, os.path.join('V://GIS//Maps//County//11x17shd//',i))
file.close()
ftp.quit()

[FrankPerks]

Sounds crazy but you have to trust me here. If you are going to use storbinary, the file cant be opened in binary mode. I know what you are going to say but I just figured this out in the last couple weeks myself and it just doesnt work.

fileList=[]
for i in os.listdir(dir):
    if i.split('.')[1] == 'pdf' #if it ends in pdf
        fileList.append(i) #put it in the list
ftp = FTP("XXX.XXX.XXX.XXX",username, password) #save a line and just put your U:P here.
for i in fileList:
    file = open(i, "r") #open in normal read mode
   ftp.cwd("//data2//ftp//pub//download//maps//")
    ftp.storbinary('STOR  %s' % i, os.path.join('V://GIS//Maps//County//11x17shd//',i)) 
    file.close()
ftp.quit()

Your using storbinary wrong:

ftp = FTP("XXX.XXX.XXX.XXX",username, password) #save a line and just put your U:P here.
for i in fileList:
    # don't use double //, this will rely on the server to properly handle it (most will)
    # also don't use a trailing slash since VAX/RSIC will try the folder "maps/"
    ftp.cwd("/data2/ftp/pub/download/maps")
    
    # ftp.storbinary takes a FILE object not a path, python will automatically close the file
    ftp.storbinary('STOR %s' % i, open(os.path.join('V://GIS//Maps//County//11x17shd//',i), "rb"))
ftp.quit()

storbinary takes a file like object as the second parameter.

If you are transfering ANYTHING but a text file, you need to use binary  otherwise you will be faced with data corruption:

From Python Documentation:

On Windows, 'b' appended to the mode opens the file in binary mode, so there are also modes like 'rb', 'wb', and 'r+b'.  Python on Windows makes a distinction between text and binary files; the end-of-line characters in text files are automatically altered slightly when data is read or written.  This behind-the-scenes modification to file data is fine for ASCII text files, but it�??ll corrupt binary data like that in JPEG or EXE files.  Be very careful to use binary mode when reading and writing such files.  On Unix, it doesn�??t hurt to append a 'b' to the mode, so you can use it platform-independently for all binary files.

There are/can be tons of other little nitpicky issues relating to FTP, is old and was not standardized properly, alot of ftp servers have confilicting behaviour, (some servers use binary as a standard mode, others use A, etc etc). In general if you are going to try and transfer a binary file i suggest you manually change the server's connection type (ftplib is flaky about this):

ftp = FTP("XXX.XXX.XXX.XXX",username, password) #save a line and just put your U:P here.
for i in fileList:
    ftp.cwd("/data2/ftp/pub/download/maps")

    # Manually tell the server to change to binary mode:
    #   I indicates Binary,
    #   A indicates text
    # There are a few other transfer modes however they are not used very often
    # just print the return message, if its an error ftplib will raise an exception
    print ftp.sendcmd("TYPE I")
    
    # ftp.storbinary takes a FILE object not a path, python will automatically close the file
    ftp.storbinary('STOR %s' % i, open(os.path.join('V://GIS//Maps//County//11x17shd//',i), "rb"))
ftp.quit()

[ChrisMathers]

It does take a file object not a path. Thank you for pointing out that typo. Where I have the path should be the variable you assigned the open(object) to.

I have had python not close a file which meant I had to restart to make windows give up its hold on the file. I suggest explicitly opening and closing your files instead of hoping python trash cleanup closes it for you.

[JesusaRomero]

Hi, this thread was forwarded to me... So, I have a script that is scheduled to run every week on Mondays and I think it is doing what you want to accomplish (just modify it a little). As mentioned in the description below, it deletes the existing folders and contents on FTP and replaces it with folders and contents from our file server (or even local if you want). We needed to transfer over multiple folders and files, so it was important to retain the same folder and file structure from one area to the other. The script then deletes the folders and contents off our server after the transfer.

I've learned that there are some limitations when working with FTP, like you can't manipulate files using certain modules in "FTP land". I also discovered that you can't copy and paste the whole folder and contents from server to FTP in one line or so, and I couldn't seem to overwrite folders/files on FTP (hence the delete first then copy and paste), so I had to create the folders on FTP and then copy the files from our server into the corresponding folder on FTP. Change everything in red. I'm moving "TIF" and "TXT" files so just change that. I have another "TXT file that is not in any folder so that one shouldn't and isn't transferred over (sorry for the confusion). Go ahead and modify for your own use. You probably don't need half the things down there anyway... Hope this helps. Like you, I'm also fairly new to Python. 😉

If anyone has suggestions for improving my script below, let me know.

# Written by Jesusa M. Romero on 06/10/10.
# Script Description:
# 1) Deletes the existing folders AND files on FTP
# 2) Lists folders on rcant7 and creates the same list of folders on FTP
# 3) Transfers TIF images from rcant7 to the corresponding folders on FTP
# 4) Deletes the list of folders on rcant7
# Note: The "notes.txt" file on rcant7 isn't transferred or deleted.
# ---------------------------------------------------------------------------#

import ftplib, os, shutil, sys

startpath = r"\\rcant7\data\MAPPING\GIS\Images"
ftp = ftplib.FTP('IP ADDRESS GOES HERE')
ftp.login('USERNAME', 'PASSWORD')
directory = '/ACR_Images/'
ftp.cwd(directory)

print "Script: Image transfer from Rcant7 to Survey FTP now executing..."
print " "

ftpdirset = ftp.nlst(directory)
for ftpd in ftpdirset:
    dirfold = directory + ftpd
    print "Visiting existing FTP folders/files: "  + dirfold
    try:
        ftp.delete(dirfold)
        print "FTP file deleted: " + dirfold
        print "-" * 70
    except:        
        ftpfiles = ftp.nlst(dirfold)
        for ftpfile in ftpfiles:
            dirfoldfile = dirfold + "/" + ftpfile
            print "     File in folder deleted: " + dirfoldfile
            ftp.delete(dirfoldfile)
        ftp.rmd(dirfold)
        print "FTP folder deleted: " + dirfold
        print "-" * 70
print "-" * 70

print "FTP directory " + directory + " has been emptied."
print "Images will be transferred."
print "-" * 70
print "-" * 70
print " "

dirListST = os.listdir(startpath)
for folder in dirListST:
    if os.path.isdir(startpath + "\\"+ folder):    
        print "Creating new " + folder + " folder in FTP " +  directory
        print "Transferring files to FTP now..."
        ftp.cwd(directory)
        ftp.mkd(folder)
        stpathfolder = os.path.join(startpath, folder)
        dirListST2 = os.listdir(stpathfolder)
        for specfile in dirListST2:                
            basename, extension = specfile.split(".")
            pathforspecfile = stpathfolder + "\\" + specfile
            ftp1 = startpath + "\\"+ folder + "\\" + specfile
            ftp.cwd(directory + folder + "/")
            if extension == "tif" or extension == "txt":
                f = open(ftp1,"rb")
                a = "STOR " + specfile
                ftp.storbinary (a, f)
                print "     Transfer completed for " + specfile
                f.close()
            else:
                print "     File not transferred: " + specfile
        print stpathfolder + " has been deleted."
        shutil.rmtree(stpathfolder)
    else:
        print "*** " + folder + " is not a directory. Not transferred from Rcant7."
    print "-" * 70
print "-" * 70
print "IMAGE TRANSFER HAS BEEN COMPLETED."
                              
ftp.quit()

If you want to discuss this further, just post your e-mail. 🙂

[PaulHuffman]

If you want to discuss this further, just post your e-mail. 🙂

I'm interested.  huffmanp@ykfp.org.  I have been trying to keep an ftp site in sync with a linux website directory and I've been going about this very inefficiently, partly because few feature requests keep being added on. I wrote a little perl script a few years ago to give users a form to upload files to a directory on the web site.  Recently I was asked if I could write a version of the script to allow uploads for a new project to a new directory, and also push these files out to an ftp account on the hosting company, so the users of this data could access the ftp directory without needing the username, password of the website and have these users pawing through all the directories on the web site to find the one they are supposed to be accessing. 

The most logical thing for me to do would be to modify the perl script to push a copy of any file uploaded to the ftp site also.  But I looked at the perl script and though "Cripes, I've forgotten everything I ever knew about perl."  I looked at Uber-Uploader since it's written in php which I can manage a bit better and has a perl component so the upload file size can be larger than with php without fiddling with php.ini settings.  But I couldn't get Uber-Uploader to run on my host.  Then I started looking at python ways to do it since I need to get better with python. 

I got python to work for me thusly:

import ftplib
import os
import sys
import traceback

print "Logging in..."
ftp = ftplib.FTP()
ftp.connect('ftphostname')
print ftp.getwelcome()
try:
    try:
        ftp.login('username', 'password')
        ftp.cwd('target directory')
        # move to the desired upload directory
        print "Currently in:", ftp.pwd()

        print "Uploading...",
        fullname = '../sourcedirectoy/filename'
        name = os.path.split(fullname)[1]
        f = open(fullname, "rb")
        ftp.storbinary('STOR ' + name, f)
        f.close()
        print "OK"
        
        print "Files:"
        print ftp.retrlines('LIST')
    finally:
        print "Quitting..."
        ftp.quit()
except:
    traceback.print_exc()

But now I'll have to see if I can make it work on a whole directory.  If I were clever, I'd run this python script every few hours as a cron job (in windows, a "scheduled task"), and test the source directory for any file newer than last time the script ran in a cron.  If I were really clever, I'd just add the ftp push to my original perl script.