Create Text File or Other Log on ArcGIS Online Notebooks
Accepted Solutions
We use a series of standalone hosted tables for our scheduled Notebook tasks. I would not attempt to use a text file. Though it's possible to use files inside of a Notebook separate from your portal's Content, you want to make sure something important like a log or auxiliary text is kept somewhere that's accessible to the Notebook, but also to other users who may need to access it. A hosted table does that.
Here are some examples:
Document ID Tables
We synchronize a workflow table with documents from the County Recorder's website every night. We can't directly access their database, so we scrape the document details from the website by iterating over the document ID. The table stores the ID of the last successfully-scraped document, so that future runs of the script can pick up where the previous day left off.
Additionally, there are times when a document's details are finalized on a future date, and may be missed initially (the HTTP request returns a 400). To handle these, we have a second table of "skipped" IDs, to be re-queried in future runs of the script, so as not to miss documents added out of sequence.
Layer Update Timestamps
We have a few hosted layers that update from SQL queries of a non-public database. Due to the size of the layers being updated, a truncate / append or overwrite process would be time-consuming and unnecessary, due to the relatively few edits that happen on a given day.
Using a hosted table, we store timestamps of the last successful update for the hosted layers. A scheduled notebook is then able to read that table, then adjust its SQL query to only return features that were edited since that timestamp.
Notebook Logs
Finally, we have another hosted table that stores custom log messages from our scheduled notebook runs. Much like a standard log file, it stores a message, log source, timestamp, and message severity code. These pipe into a Dashboard List element, allowing me to quickly and easily look through recent logs and see errors and warnings right away.
The notebooks simply access the log table as any other layer, and use edit_features(adds=[...]) to add messages.
We use a series of standalone hosted tables for our scheduled Notebook tasks. I would not attempt to use a text file. Though it's possible to use files inside of a Notebook separate from your portal's Content, you want to make sure something important like a log or auxiliary text is kept somewhere that's accessible to the Notebook, but also to other users who may need to access it. A hosted table does that.
Here are some examples:
Document ID Tables
We synchronize a workflow table with documents from the County Recorder's website every night. We can't directly access their database, so we scrape the document details from the website by iterating over the document ID. The table stores the ID of the last successfully-scraped document, so that future runs of the script can pick up where the previous day left off.
Additionally, there are times when a document's details are finalized on a future date, and may be missed initially (the HTTP request returns a 400). To handle these, we have a second table of "skipped" IDs, to be re-queried in future runs of the script, so as not to miss documents added out of sequence.
Layer Update Timestamps
We have a few hosted layers that update from SQL queries of a non-public database. Due to the size of the layers being updated, a truncate / append or overwrite process would be time-consuming and unnecessary, due to the relatively few edits that happen on a given day.
Using a hosted table, we store timestamps of the last successful update for the hosted layers. A scheduled notebook is then able to read that table, then adjust its SQL query to only return features that were edited since that timestamp.
Notebook Logs
Finally, we have another hosted table that stores custom log messages from our scheduled notebook runs. Much like a standard log file, it stores a message, log source, timestamp, and message severity code. These pipe into a Dashboard List element, allowing me to quickly and easily look through recent logs and see errors and warnings right away.
The notebooks simply access the log table as any other layer, and use edit_features(adds=[...]) to add messages.
