Select to view content in your preferred language

Notebook python output different than stand-alone python script

162
6
2 weeks ago
ShariF
by
Regular Contributor

Context: Our organization has developed a python script in Jupyter Notebooks that scrapes an electricity providers website and compares the power outages with some of our at risk residents' addresses. If there is a power outage then it uses an email function to alert staff of those power outages. We want to schedule this script as a stand-alone script on a local server with task scheduler to run every hour. 

^^^ Since there are currently no power outages, I updated the script to email us all those residences that have power on. (Should be all of our at risk population). This works beautifully in Jupyter Notebooks.

Issue: When I export the notebook as a stand-alone python script and run it in PyCharm, it doesn't provide the same results. During the test for those with power on it shows zero and we do not get sent an email. 

Example results in Jupyter Notebooks creating a list: (the list is called "off_records" but it's really grabbing "Power_Status" == "ON" for testing purposes)

Residents Power Outage Jupyter Notebooks example.png

 

This is the result of the same exact script using PyCharm:

ShariF_0-1755275611889.png

 

Help! Any reason why these results would be different? 

Thanks for your time, 

Shari

0 Kudos
6 Replies
AustinAverill
Frequent Contributor

It's apprent that from PyCharm, the variable "match" is not being evaluated as true. Unfortunately, we cannot really debug the cariable since its referencing something that is outside of what you provided.

 

Add more print statements to help you debug why your "if match" conditions is not being met.

0 Kudos
ShariF
by
Regular Contributor

Thanks for pointing that out. I'm not sure how to share this script with you to see it all as it has sensitive data within it. 

0 Kudos
AustinAverill
Frequent Contributor

Understandable. Unfortunately, without much more context, it's going to be difficult to offer any more advice beyond adding in more print statements for debugging. And double check the request you are making to the service you're scraping for data to ensure that it hasn't been altered from the notebook.

0 Kudos
ShariF
by
Regular Contributor

The layer with the sensitive info is locked not shared publicly but here's the script anyway. If something stands out to you please let me know. 

"""Script checks Oncor API and compares with an existing layer of COR STEAR residents.
Scrapes Oncor website that includes power status. If power status is off then an email is sent to oem.
Written in AGOL Notebooks by Ryan Steel. Copied from there and edited by Shari on 08/14/2025.
Python 3"""

from arcgis.gis import GIS
import requests
from tqdm import trange
import smtplib
from datetime import datetime

def emailAlert(toList, subject, body):
    email_user = 'noreplay@cor.gov'
    message = 'Subject: {}\n\n{}'.format(subject, body)
    server = smtplib.SMTP('outbound.cor.gov', 25)
    server.sendmail(email_user, toList, message)
    server.close()

try:

    start_time = time.time()

    # 1. Connect to your GIS
    gis = GIS("home")

    # 2. Connect to feature layer
    print("Connecting to feature layer...")
    layer_item = gis.content.get("c8018b05abc74086a18c95ff1bd77657")
    layer = layer_item.layers[0]

    # 3. Query all features (return all fields)
    print("Querying existing features...")
    features = layer.query(
        where="1=1",
        out_fields="*",
        return_geometry=False
    ).features
    print(features)

    # 4 and 5 Create mapping of ESI to FID and Get list of ESIs from the layer
    esi_to_fid = {
        str(f.attributes["ESI"]).strip(): f.attributes["FID"]
        for f in features
        if f.attributes["ESI"] is not None
    }

    esi_list = list(esi_to_fid.keys())
    print(esi_list)

    # 6. Fetch power status and prepare updates
    print("Fetching power status and preparing updates...")
    updates = []
    errors = []

    current_date = datetime.now().strftime('%Y-%m-%d')  # Get the current date in the desired format

    for esi in trange(222, desc="Checking ESI power status"):
        try:
            url = f"https://www.oncor.com/outages/api/customerOutage/outage/checkStatus?esiid={esi}&source=ORS"
            response = requests.get(url)
            response.raise_for_status()
            data = response.json()

            # ✅ Correct JSON parsing
            api_status = (
                data.get("getOutageStatusResponse", {})
                    .get("powerStatus", {})
                    .get("status", "")
                    .strip()
                    .upper()
            )
            final_status = "OFF" if api_status == "OFF" else "ON"

            fid = esi_to_fid.get(esi)
            if fid is not None:
                updates.append({
                    "attributes": {
                        "FID": fid,
                        "Power_Status": final_status,
                        "Last_Updated": current_date  # Update the Last_Updated field
                    }
                })
            else:
                errors.append((esi, "FID not found"))

        except Exception as e:
            errors.append((esi, str(e)))

    # 7. Apply updates to feature layer
    print(f"Updating {len(updates)} features in the layer...")
    if updates:
        result = layer.edit_features(updates=updates)
        print("Update complete.")
    else:
        print("No updates to apply.")

    # 8. Compile OFF records for email
    print("Compiling OFF records for notification...")
    off_records = []

    for update in updates:
        if update["attributes"]["Power_Status"] == "OFF":
            fid = update["attributes"]["FID"]
            match = next((f for f in features if f.attributes["FID"] == fid), None)
            if match:
                attr = match.attributes
                full_name = f"{attr.get('First_Name', '')} {attr.get('Last_Name', '')}".strip()
                address = f"{attr.get('Street_Address_1', '')} {attr.get('Apt_Suite_Unit_1', '')}, {attr.get('City_1', '')}, {attr.get('State_Province_1', '')} {attr.get('Postal_Code_1', '')}".strip()
                phone = attr.get("Phone_1", "N/A")
                off_records.append(f"Name: {full_name}\nAddress: {address}\nPhone: {phone}\n")
    print(off_records)

    # 9. Send email if any power OFF cases
    # Convert off_records to a single string for the email body
    email_body = "\n\n".join(off_records)

    # Specify the recipients of the email
    to_list = ['shari.forbes@cor.gov', 'ryan.steele@cor.gov']  # Replace with actual recipient emails
    subject = f"Power OFF Notifications - {datetime.now().strftime('%Y-%m-%d')}"

    # Send the email if there are any off_records
    if off_records:
        print("Sending email notification...")
        emailAlert(to_list, subject, email_body)
        print("Email sent.")
    else:
        print("No power OFF records to notify.")

    end_time = time.time()
    elapsed_time = ((end_time - start_time) / 60)
    elapsed_timeR = round(elapsed_time, 2)

    print("SUCCESS!!")

    # Send success email alert
    emailAlert(['shari.forbes'],
    subject = 'STEAR Power Status python code success',
    body = 'STEAR Power STatus python code ran successfully in ' + str(elapsed_timeR) + ' minutes.\n yay!')

except Exception as e:
    print("Error: " + e.args[0])

 

0 Kudos
AustinAverill
Frequent Contributor

So, i think the issue is with line 104:

match = next((f for f in features if f.attributes["FID"] == FID, None))

My guess would be that you are not receiving any matches in this variable assignment, and therefor match is always evaluating to None and not executing your `if match:` condition. You could add a print statement after line 104 to to get what data is being assigned to the match variable and dive into why it isn't finding the match in your features data. You may also want to verify your data is matching correctly for FID - as in that they are both the same data type (string to string, int to int, etc.)

0 Kudos
ShariF
by
Regular Contributor

THank you. I'll look into it. 

0 Kudos