topic Re: Python Time Modification Help in Python Questions

Python Time Modification Help

SamLee1 — Fri, 27 Dec 2019 16:39:37 GMT

Hello,

When I run my python script, it downloads a file from a website to a directory and it moves an old file from that directory to an archived folder and renames it with filename+date moved/date when script runs.

I am trying to fix it to make it show filename+modified/created date instead of filename+date moved/date when script runs.

This file is in the archived folder by the way.

Attached is the python code.

Re: Python Time Modification Help

JoeBorgione — Fri, 27 Dec 2019 17:19:07 GMT

I'd be happy to look at your script, but downloading a zip file makes our security team crazy. If you could just use the syntax highlighter and paste it there, that would be great....

Re: Python Time Modification Help

SamLee1 — Sat, 11 Dec 2021 11:00:52 GMT

#!/usr/bin/env python
import requests
import os.path
import sys
import shutil
import pandas as pd
from datetime import datetime

def getFile():
    url = "https://www.gosolarcalifornia.ca.gov/equipment/documents/Grid_Support_Inverter_List_Full_Data.xlsm"
    content = requests.get(url)
    date = datetime.date(datetime.now())
    directory = "C:\Users\slee2\Folder"
    archivedDirectory = "C:\Users\slee2\Folder\ArchivedFolder"
    #date = datetime.datetime.fromtimestamp(os.path.getmtime(archivedDirectory)))
    fileName = "Grid_Support_Inverter_List_Full_Data"
    extension = ".csv"
    path = fileName + extension
    print ("Files will be saved in the following directory: "+directory+"\n")
    print ("Old Files with the same name will be archived in the following directory: "+archivedDirectory+"\n")
    print ("NOTICE: File will be converted to .csv file...")
    if os.path.isdir(directory):
        #Check if the Archived Folder Exists, if it does not create it
        if(os.path.isdir(archivedDirectory)):
            pass
        else:
            os.mkdir(archivedDirectory)
        print ("SUCCESS: Directory was found.")
        counter = 0
        if (os.path.exists(directory+"\\"+fileName+extension)):
            counter += 1
            while (os.path.exists(archivedDirectory+"\\"+fileName+"_%s_%s"%(str(date), str(counter))+extension)):
                counter += 1
        if counter > 0:
            print ("File: "+fileName+" already exists in directory,moving old file to Archived Folder. ")
            shutil.move(directory+"\\"+fileName+extension, archivedDirectory+"\\"+fileName+"_%s"%(str(date))+extension)
        with open(os.path.join(directory, path), 'wb') as f:
            f.write(content.content)
        deleteRows(directory, path)
        print("Done.")
    else:
        print("ERROR: Directory is not found.")
        sys.exit()
def deleteRows(directory, path):
    newFile = pd.read_excel(os.path.join(directory,path))
    newFile = newFile.drop(range(14), axis = 0)
    newFile.to_csv(os.path.join(directory, path), encoding='utf8', header=False, index=False)
def main():
    getFile()

if __name__ == "__main__":
    main()   
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Sorry I didn't know how to paste code into these and now I do, thanks.

Re: Python Time Modification Help

JoeBorgione — Fri, 27 Dec 2019 17:54:11 GMT

Take a look here: How to get file creation & modification date/times in Python? - Stack Overflow (I just googled 'extract date modified from windows file in python' to find it and others...)

Re: Python Time Modification Help

SamLee1 — Fri, 27 Dec 2019 17:57:57 GMT

Hello Joe,

Yes I used the exact same link as you and still didn't get anything.

I stored it into a variable and it didn't work.

Re: Python Time Modification Help

JoeBorgione — Fri, 27 Dec 2019 18:04:55 GMT

Well, shoot! Thought that would get you going. There are a couple guys out there lurking that probably have a solution for you...

Re: Python Time Modification Help

RandyBurton — Sat, 11 Dec 2021 11:00:55 GMT

In your code at line 9, you are getting the current date/time and using it in lines 32 and 36. I'm not sure your code inside the loop starting at line 22 is doing what you expect. But for now, just focusing on the date issue, I would suggest trying something like this to replace lines 29-36:

if (os.path.exists(directory+"\\"+fileName+extension)):
     modDate = datetime.fromtimestamp(os.path.getmtime(directory+"\\"+fileName+extension))
     shutil.move(directory+"\\"+fileName+extension, archivedDirectory+"\\"+fileName+str(modDate)[:10]+extension)‍‍‍

Back to the loop, at line 32, do you want to check if multiple files exist that have the current date in the file name?

Hope this helps.

Re: Python Time Modification Help

SamLee1 — Sat, 11 Dec 2021 11:00:57 GMT

Hello Randy,

I get an error from doing that. Been stuck on this issue for a long time, used every single online resource out there and still couldn't fix the issue.

The error only occurs if I already have a file in the folder directory.

Code:

#!/usr/bin/env python
import requests
import os.path
import sys
import shutil
import pandas as pd
from datetime import datetime
import time

def getFile():
	url = "https://www.gosolarcalifornia.ca.gov/equipment/documents/Grid_Support_Inverter_List_Full_Data.xlsm"
	content = requests.get(url)
	date = datetime.date(datetime.now())
	directory = "C:\\Users\\slee5\\Desktop\\Folder"
	archivedDirectory = "C:\\Users\\slee5\\Desktop\\Folder\\ArchivedFolder"
	#date = datetime.datetime.fromtimestamp(os.path.getmtime(archivedDirectory)))
	fileName = "Grid_Support_Inverter_List_Full_Data"
	extension = ".csv"
	path = fileName + extension
	print ("Files will be saved in the following directory: "+directory+"\n")
	print ("Old Files with the same name will be archived in the following directory: "+archivedDirectory+"\n")
	print ("NOTICE: File will be converted to .csv file...")
	if os.path.isdir(directory):
		#Check if the Archived Folder Exists, if it does not create it
		if(os.path.isdir(archivedDirectory)):
			pass
		else:
			os.mkdir(archivedDirectory)
		print ("SUCCESS: Directory was found.")
		counter = 0
		if (os.path.exists(directory+"\\"+fileName+extension)):
			counter += 1
			while (os.path.exists(archivedDirectory+"\\"+fileName+"_%s_%s"%(str(date), str(counter))+extension)):
				counter += 1
		if counter > 0:
			print ("File: "+fileName+" already exists in directory,moving old file to Archived Folder. ")
			shutil.move(directory+"\\"+fileName+extension, archivedDirectory+"\\"+fileName+"_%s"%(str(date))+extension)
			path1 = archivedDirectory+"\\"+fileName+"_%s"%(str(date))+extension
			date1 = datetime.fromtimestamp(os.path.getmtime(path1))
			date1 = str(date1)
			date1 = date1.strftime('%y-%m-%d')
			os.rename(path1,archivedDirectory+"\\"+fileName+"_%s"%(str(date1).replace(":"," "))+extension)
		with open(os.path.join(directory, path), 'wb') as f:
			f.write(content.content)
		deleteRows(directory, path)
		print("Done.")
	else:
		print("ERROR: Directory is not found.")
		sys.exit()
def deleteRows(directory, path):
	newFile = pd.read_excel(os.path.join(directory,path))
	newFile = newFile.drop(range(14), axis = 0)
	newFile.to_csv(os.path.join(directory, path), encoding='utf8', header=False, index=False)
def main():
	getFile()

if __name__ == "__main__":
	main()   
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Here is some background information for the script:

Re: Python Time Modification Help

RandyBurton — Sat, 11 Dec 2021 11:01:00 GMT

Try replacing lines 39-42 in your latest code with (maintaining the indentation):

               date1 = datetime.fromtimestamp(os.path.getmtime(path1))
               # date1 = str(date1) # delete or comment out - this line is not needed and is causing the error
               date1 = date1.strftime('_%y-%m-%d') # this converts datetime object to a formatted string - add and underscore to format
               os.rename(path1,archivedDirectory+"\\"+fileName+date1+extension) # previous line formats date1 to text‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The os.path.getmtime returns a timestamp which is converted by datetime.fromtimestamp into a datetime object ( year, month, day, hours, minutes, seconds). Since date1 is now a datetime object, strftime can be used to format it into a string "_yy-mm-dd". Once it is a string, it can be added into the file's name sequence: fileName+date1+extension.

In my previous code snippet, I was renaming the file as it was being moved. In this snippet the file is renamed after it was moved. Either way should work for you.

Re: Python Time Modification Help

SamLee1 — Sat, 11 Dec 2021 11:01:03 GMT

Hello Randy,

Thanks so much for the help, been stuck on this for a week.

One small error, how can I change 19-12-30 to 2019-12-30?

Code In Case:

#!/usr/bin/env python
import requests
import os.path
import sys
import shutil
import pandas as pd
from datetime import datetime
import time

def getFile():
	url = "https://www.gosolarcalifornia.ca.gov/equipment/documents/Grid_Support_Inverter_List_Full_Data.xlsm"
	content = requests.get(url)
	date = datetime.date(datetime.now())
	directory = "C:\\Users\\slee5\\Desktop\\Folder"
	archivedDirectory = "C:\\Users\\slee5\\Desktop\\Folder\\ArchivedFolder"
	#date = datetime.datetime.fromtimestamp(os.path.getmtime(archivedDirectory)))
	fileName = "Grid_Support_Inverter_List_Full_Data"
	extension = ".csv"
	path = fileName + extension
	print ("Files will be saved in the following directory: "+directory+"\n")
	print ("Old Files with the same name will be archived in the following directory: "+archivedDirectory+"\n")
	print ("NOTICE: File will be converted to .csv file...")
	if os.path.isdir(directory):
		#Check if the Archived Folder Exists, if it does not create it
		if(os.path.isdir(archivedDirectory)):
			pass
		else:
			os.mkdir(archivedDirectory)
		print ("SUCCESS: Directory was found.")
		counter = 0
		if (os.path.exists(directory+"\\"+fileName+extension)):
			counter += 1
			while (os.path.exists(archivedDirectory+"\\"+fileName+"_%s_%s"%(str(date), str(counter))+extension)):
				counter += 1
		if counter > 0:
			print ("File: "+fileName+" already exists in directory,moving old file to Archived Folder. ")
			shutil.move(directory+"\\"+fileName+extension, archivedDirectory+"\\"+fileName+"_%s"%(str(date))+extension)
			path1 = archivedDirectory+"\\"+fileName+"_%s"%(str(date))+extension
			date1 = datetime.fromtimestamp(os.path.getmtime(path1))
			#date1 = str(date1) # delete or comment out - this line is not needed and is causing the error
			date1 = date1.strftime('_%y-%m-%d') # this converts datetime object to a formatted string - add and underscore to format
			os.rename(path1,archivedDirectory+"\\"+fileName+date1+extension) # previous line formats date1 to text
		with open(os.path.join(directory, path), 'wb') as f:
			f.write(content.content)
		deleteRows(directory, path)
		print("Done.")
	else:
		print("ERROR: Directory is not found.")
		sys.exit()
def deleteRows(directory, path):
	newFile = pd.read_excel(os.path.join(directory,path))
	newFile = newFile.drop(range(14), axis = 0)
	newFile.to_csv(os.path.join(directory, path), encoding='utf8', header=False, index=False)
def main():
	getFile()

if __name__ == "__main__":
	main()   
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Python Time Modification Help

RandyBurton — Thu, 02 Jan 2020 07:46:57 GMT

To change to a 4 digit year, just use an upper case "Y" in line 41:

date1 = date1.strftime('_%Y-%m-%d')‍

See strftime and strptime behavior for the datetime format codes.

Re: Python Time Modification Help

SamLee1 — Thu, 02 Jan 2020 17:33:23 GMT

Sweet worked.

Can I basically use that link to change the formatting of the date to any combination of date formats?

For example, change it to 12-30-2019 and etc.?

Also, how can I mark this discussion/post as done/correct?

Re: Python Time Modification Help

RandyBurton — Thu, 02 Jan 2020 18:50:44 GMT

The link should show you all the formatting codes related to the timedate object. To display the date with month first and year last, just move %Y to the end.

To mark a correct answer, you would need to change this "discussion" to a "question". Then a correct answer option would be available.

Re: Python Time Modification Help

RandyBurton — Sat, 11 Dec 2021 11:01:06 GMT

Since you are interested in file creation/modification dates, I was experimenting with the following:

from datetime import datetime
import dateutil.tz as tz  # to convert utc to local time
import dateutil.parser as parser # will parse a variety of datetime formats
import requests
import  os.path
import xml.etree.ElementTree as ET # for parsing xml file
import zipfile # to open zip file (excel uses this format)

directory = r'C:\Path\To\Save\Directory' # directory for saved file
file = 'Grid_Support_Inverter_List_Full_Data.xlsm'  # name of saved file

# file to download
url = "https://www.gosolarcalifornia.ca.gov/equipment/documents/Grid_Support_Inverter_List_Full_Data.xlsm"
content = requests.get(url)
# print content.headers # for debugging

# get Last-Modified date from header and print as a local time
webModDate = parser.parse(content.headers['Last-Modified']).astimezone(tz.tzlocal())
print "Web Last-Modified: {}".format(webModDate.strftime('%Y-%m-%d %H:%M:%S'))

# save downloaded excel file to local drive
with open(os.path.join(directory, file), 'wb') as f:
    f.write(content.content)
f.close()

# excel file format is a type of zip file
# read an xml file inside the excel file to get file modified date
# and convert to local date timetime
zip = zipfile.ZipFile(os.path.join(directory, file))
props = zip.open('docProps/core.xml') # core document properties
xmlText = props.read()
zip.close()
# print xmlText # for debugging

ns = { 'cp' : 'http://schemas.openxmlformats.org/package/2006/metadata/core-properties',
       'dc' : 'http://purl.org/dc/elements/1.1/',
       'dcterms' : 'http://purl.org/dc/terms/' }

tree = ET.fromstring(xmlText)

creator =  tree.find('dc:creator', ns).text
lastModBy =  tree.find('cp:lastModifiedBy', ns).text
created =  tree.find('dcterms:created', ns).text
modified = tree.find('dcterms:modified', ns).text
# print creator, lastModBy, created, modified # for debugging

# convert excel modified date to local datetime
xlModDate = parser.parse(modified).astimezone(tz.tzlocal())
print "Excel file modified: {}".format(xlModDate.strftime('%Y-%m-%d %H:%M:%S'))

# get the saved file's create and modified date from operating system
# datetime will be timestamp in local time, when file was saved
# (basically the time when this script was run)
osCreateDate = datetime.fromtimestamp(os.path.getctime(os.path.join(directory, file)))
print "OS file created: {}".format(osCreateDate.strftime('%Y-%m-%d %H:%M:%S'))
osModDate = datetime.fromtimestamp(os.path.getmtime(os.path.join(directory, file)))
print "OS file modified: {}".format(osModDate.strftime('%Y-%m-%d %H:%M:%S'))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The results:

Web Last-Modified: 2020-01-02 10:48:55
Excel file modified: 2020-01-02 09:32:50
OS file created: 2020-01-03 18:12:12
OS file modified: 2020-01-03 18:12:12‍‍‍‍

From this, you can see that the Excel file was modified a bit over an hour before it was put on the web server. The modified date/time came directly from xml inside the Excel file.

The operating system gave the file the same create and modified date indicating when the file was downloaded and saved to the local system (and not the actual time the Excel file was last modified).

You should be able to use requests.head(url) to retrieve just the header to examine the modified date and then determine if you want to actually download the file.

Re: Python Time Modification Help

SamLee1 — Sat, 04 Jan 2020 19:28:18 GMT

Thank you for further information and coding about this, very beneficial!