ArcGIS online UTC Time Zone converting to local timezone

14040
34
Jump to solution
11-21-2019 06:44 PM
YashKaja
New Contributor II

We have few DATETIME columns stored in UTC timezone in database. When we view them in ArcGIS online, the timestamps are converted to local timezone, thereby causing the confusion with dates.

"The date and time in a pop-up do not match the date and time in the underlying data."

Followed below articles suggested by support. But no luck. Any thoughts on this issue please? Thanks in advance!

How To: Set the UTC time field to follow a specific time zone 

https://community.esri.com/thread/205882-date-changed-to-strange-values-in-feature-class-when-upload... 

0 Kudos
34 Replies
by Anonymous User
Not applicable

Hi Peter,

So I had misspoke before... my time shows correctly when manually running the overwrite from Pro and selecting the proper timezone in the configuration settings. When I'm overwriting via the script and unable to set the timezone it's actually considering my data as UTC and converting it to the local time. So it's "double tapping" the timezone conversion and setting everything back -5 hours for CDT (-6 for CST).

I've read both of the documentation pages you linked, much appreciated by the way... but was unable to find any clear answer. Perhaps I missed something.

On the publish module it specifically describes Publish_parameters arguments as "Optional dictionary. containing publish instructions and customizations. Cannot be combined with overwrite" and on the Publish Item-ArcGIS Rest API the last thing said about service definition publish parameters is "

Note:

The publishParameters properties are optional when publishing a service definition (.sd), vector tile package (.vtpk), or tile package (.tpk) file." and then it goes on to tell you how to write for csv, excel, shapefile, gdb, etc.

I have my sd draft definition clearly laid out...

arcpy.mp.CreateWebLayerSDDraft( mp, sddraft, sd_fs_name, 'MY_HOSTED_SERVICES', service_type = 'FEATURE_ACCESS', folder_name = 'BRPD_Hosted_Layers', overwrite_existing_service = True, copy_data_to_server = True, enable_editing = False, allow_exporting = True)

I just wish there was a more straight forward way of specifying what timezone my data is in when overwriting using a script. 

Currently, the "work around" we've decided on going with is to have a Central time zone field and a UTC time field for each of the 5 date fields connected to our data. This will allow the time to be used in Central time using the Central Time field when working in Pro/ Desktop and used in Central Time when using the UTC field once hosted online.

 ¯\_(ツ)_/¯

0 Kudos
by Anonymous User
Not applicable

Hi Brandon, thanks for the response. My fault on forgetting that the publish parameters dictionary cannot work with overwrite. I'm happy to hear there is a workaround for now but I will dig into the question of an easier way to specify time zone when overwriting data and see what I can find/get an enhancement logged. 

by Anonymous User
Not applicable

Thanks Peter, I'd be really interested in seeing what you come up with.

by Anonymous User
Not applicable

Hi Peter,

I wanted to let you know that I ran across another issue I believe is tied to overwrite process being unable to pass publish parameters... or... use existing rather than overwrite to an arbitrary default.

Below are a couple of screenshots showing some field settings of the layer I am overwriting with the script. Some of these fields, such as CAD ID Num, Case Num, Call Date, Primary Officer as a few examples, come directly from the CAD system data feed. Allowing nulls is understandable in some cases ( a case number is not generated for every CAD call  so some nulls will exist), and others are negatable (this data view pullls the last 180 days of data, if a call happened  before or after time existed {outside of the date/ time continuum} than it's kind of outside of our jurisdiction anyway haha). Hour of Call and DOW are built from the Call Date field, which allows null values but wouldn't exist if it were null... so they can allow nullls also without ever really noticing.

Highlighted in yellow are values that were causing some issues down stream when null values where allowed. When a call is closed a "final code/type" should be known upon completion... even if it is cancelled or deferred to another agency. Response time and on-scene time became an issue when creating dashboards and automating some industry standard metrics such as average response time... null values don't represent a "0" value and averaging wasn't performing properly.

 The 911 data is not shared with anyone for editing ever (in a GIS environment as a view layer extracted for a SQL database, Dispatchers do their thing in the CAD system) 

Excel document outlining the schema for the layer and how to handle problematic null values.

Once published using the overwrite script, many of these settings appear to be reverting... even one's that I specified in my sd draft definition.

For Call Date:

In Pro, I can't edit the layer and in my sd draft I have  enable_editing = False, allow_exporting = True)... surely the allow exporting is enabled.....

Oooohhhhh 

The same is true for all fields, it just defaulted to yes and only the field type and length seems to have actually passed correctly.

The response time field I mentioned 

Thank goodness any nulls that did exist were properly handled before this stage and it is a non-issue for the purposes we have.

Hope this can help in some way or another.

Thanks, 

B

by Anonymous User
Not applicable

Hi Brandon - thanks for the details here. See my responses below. Just to note, I did my testing using the FeatureSharingDraft class, which replaces the legacy CreateWebLayerSDDraft function. This may explain the differences seen with #1. 

1) Null values and edit settings not being respected:

-I wasn't able to reproduce this, though I was testing from a FGDB instead of a view layer extracted from a SQL server database. In my case, both allow nulls and editable/not editable were respected: 

Pro 2.6.2:

Online:

Below shows AnotherNoNullFld:

I would recommend creating a Support case to have an analyst take a closer look at the data and see if they can set up a repro case. It's possible that this was also something that was resolved in the FeatureSharingDraft class. 

2) Publishing with the option to allow export: 

I reproduced this using the FeatureSharingDraft class, then after some quick research realized it was logged as a bug: BUG-000117831: The export capability is not honored when using the .. 

Let me know, and I can attach your account to this bug. It's marked as In Product Plan. 

3) Time zones when overwriting web layer from Pro:

This issue was logged as an enhancement earlier this year, and it turns out specifying the time zone and daylight savings time is possible by using Python to edit the XML of the sddraft file. Someone on the Pro team (I think) wrote a code sample that was listed on the enhancement; I tested it and it works great. The sample is for FeatureSharingDraft but the XML editing part could probably be incorporated for use with CreateWebLayerSDDraft as it just edits the sddraft file. Script is below:

#### FeatureSharingDraft w/ TimeZone
#Only modify variables in the main() function

import arcpy
import os
import xml.dom.minidom as DOM
import codecs
import xml.etree.ElementTree as ET


# Function to enable extensions
def enable_extensions(sddraftPath, soe):
    # Read the sddraft xml.
    doc = DOM.parse(sddraftPath)

    # Find all elements named TypeName. This is where the server object extension
    # (SOE) names are defined.
    typeNames = doc.getElementsByTagName('TypeName')
    for typeName in typeNames:
        # Get the TypeName we want to enable.
        if typeName.firstChild.data == soe:
            extension = typeName.parentNode
            for extElement in extension.childNodes:
                    # Enable Feature Access.
                    if extElement.tagName == 'Enabled':
                        extElement.firstChild.data = 'true'
                        # Write to sddraft.
                        f = open(sddraftPath, 'w')
                        doc.writexml(f)
                        f.close()


# Function to configure properties of an extension
# soe = extension for which properties have to be added
def enable_configproperties(sddraftPath, soe, property_set):
    # Read the sddraft xml.
    doc = DOM.parse(sddraftPath)

    # Find all elements named TypeName. This is where the server object extension
    # (SOE) names are defined.
    typeNames = doc.getElementsByTagName('TypeName')
    for typeName in typeNames:
        # Get the TypeName we want to enable.
        if typeName.firstChild.data == soe:
            extension = typeName.parentNode
            # prp = extension.childNodes.getElementsByTagNameNS('PropertyArray')
            for extElement in extension.childNodes:
                if extElement.tagName == 'Definition':
                    for definition in extElement.childNodes:
                        if definition.tagName == 'ConfigurationProperties':
                            for config_prop in definition.childNodes:
                                if config_prop.tagName == 'PropertyArray':
                                    for prop in property_set:
                                        prop_set = doc.createElement("PropertySetProperty")
                                        attr = doc.createAttribute("xsi:type")
                                        attr.value = "typens:PropertySetProperty"
                                        prop_set.setAttributeNode(attr)

                                        prop_key = doc.createElement("Key")
                                        txt = doc.createTextNode(prop["key"])
                                        prop_key.appendChild(txt)
                                        prop_set.appendChild(prop_key)

                                        prop_value = doc.createElement("Value")
                                        attr = doc.createAttribute("xsi:type")
                                        attr.value = "xs:string"
                                        prop_value.setAttributeNode(attr)
                                        txt = doc.createTextNode(prop["value"])
                                        prop_value.appendChild(txt)
                                        prop_set.appendChild(prop_value)

                                        config_prop.appendChild(prop_set)

                                        # Write to sddraft
                                        f = open(sddraftPath, 'w')
                                        doc.writexml(f)
                                        f.close()

# main()
if __name__ == "__main__":

    #Set variables
    username = "your_username"
    pw = "your_password"
    outdir = r"C:\...."
    service = "ArcGIS Online Hosted Feature Service Name"
    pro_project = r"C:\...\your_project_name.aprx"

    #Overwrite the output
    arcpy.env.overwriteOutput = True
    
    # Connecting to the target portal
    arcpy.SignInToPortal("https://arcgis.com", username, pw)

    # Set output file names
    sddraft_filename = service + ".sddraft"
    sddraft_output_filename = os.path.join(outdir, sddraft_filename)

    #Access maps in the Pro project
    aprx = arcpy.mp.ArcGISProject(pro_project)
    m = aprx.listMaps()[0]
    # use if you want to share selected layers
    # layers = m.listLayers()
    # lyr = layers[0]

    
    #Specific to on-premise Enterprise deployements
    #in_server = "federated server url"
    #sddraft.federatedServerUrl = in_server
    #sddraft.copyDataToServer = False
    #sddraft.exportToSDDraft("location and name of sddraft")

    #Online - update summary through use limitations with your strings
    sddraft = m.getWebLayerSharingDraft("HOSTING_SERVER", "FEATURE", service)
    sddraft.summary = "My Summary"
    sddraft.tags = "My Tags"
    sddraft.description = "My Description"
    sddraft.credits = "My Credits"
    sddraft.useLimitations = "My Use Limitations"
    sddraft.overwriteExistingService = True

    # Create Service Definition Draft file
    sddraft.exportToSDDraft(sddraft_output_filename)

    # Set timezone - obtain valid dateFieldsTimezoneID values with arcpy.time.ListTimeZones()
    property_set = [{
    "key": "dateFieldsRespectsDayLightSavingTime",
    "value": "true"
    },
    {
    "key": "dateFieldsTimezoneID",
    "value": "Pacific Standard Time"
    }]
    # To set time zone on hosted feature service, soe = "FeatureServer"
    enable_configproperties(sddraftPath=sddraft_output_filename, soe="FeatureServer", property_set=property_set)

    # Enable extensions on server
    enable_extensions(sddraftPath=sddraft_output_filename, soe="FeatureServer")
    # enable_extensions(sddraftPath, "VersionManagementServer")
    # enable_extensions(sddraftPath, "LRServer")

    # Stage Service
    sd_filename = service + ".sd"
    sd_output_filename = os.path.join(outdir, sd_filename)
    arcpy.StageService_server(sddraft_output_filename, sd_output_filename)

    # Share to portal
    print("Uploading Service Definition...")
    arcpy.UploadServiceDefinition_server(sd_output_filename, "My Hosted Services")


I hope this helps,

-Peter 

cc Nicole Corbin‌, Aaron Falk

by Anonymous User
Not applicable

Thanks Peter, I'll be trying these out.

And yes, please attach my account.

by Anonymous User
Not applicable

Hi Peter,

The FeatureSharingDraft is working to carry over the proper timezone...

Now I'm unable to pass sharing parameters (is public, with org, with groups).

I was hoping it would be as easy as including it with the property_set along with the timezoneID...

 # Set timezone - obtain valid dateFieldsTimezoneID values with arcpy.time.ListTimeZones()
property_set = [{
"key": "dateFieldsRespectsDayLightSavingTime",
"value": "true"
},
{
"key": "dateFieldsTimezoneID",
"value": "Central Standard Time"
},
{
"key": "PackageUnderMyOrg",
"value": "true"
},
{
"key": "PackageIsPublic",
"value": "true"
}]
# To set time zone on hosted feature service, soe = "FeatureServer"
enable_configproperties(sddraftPath=sddraft_output_filename, soe="FeatureServer", property_set=property_set)

but the sharing still reverts to not shared.

 

I'm going to do a little more digging and play around with it but was hoping you may know of an answer already.

 

Thanks,

Brandon

 

 

0 Kudos
by Anonymous User
Not applicable

Hi @Anonymous User - thanks for the response and sorry for my late follow up. Figuring out the new Esri Community notifications and had missed these. My bad. I'll take a look and see if I can find an answer - I don't know one off the top of my head unfortunately. 

Thanks,

-Peter 

0 Kudos
by Anonymous User
Not applicable

Hi @Anonymous User sorry for the delayed post. Have you seen this recent GeoNet thread? It looks like the sharing settings would need to be done on the feature layer after it's published, but can be done from the same script. 

Hope this helps,

-Peter 

0 Kudos
by Anonymous User
Not applicable
Hi Peter,
No worries, better late than never as the saying goes...
I did happen upon that thread during my perusing the intrawebs. It was helpful, but a lot of things were "helpful" but never really "the solution".
Having older servers that can't update beyond 10.4 presented a unique challenge and what wound up working for me was a blended script that pulled from the CreateWebLayer SdDraft method and threw in some of the Feature Share functionality. The result is a scheduled task that automates a feature layer overwrite while allowing me to update/maintain the sharing settings AND set the timezone property accordingly.
Bingo-bango!!! Easy day lol
I do appreciate you're following up on this and I do like the idea of "post-publishing" tasks (like the share settings you mentioned) and thought heck, why not some "pre-publishing" in there as well?

Thanks again!