Hello!
I am working on an ArcGIS Online map that utilizes a CSV file, and I am wanting the CSV file to automatically update every hour or so. The CSV file originates from a website that uses a view from an SQL database; I'm using SSMS, if that is of any importance. I would connect via URL, but the issue is that there's not a link for it from the website, as it is a button-trigger activation for the CSV file. (Unless there's some way to manipulate the link to do so for me? Seems like that would be the easiest option for connecting directly to ArcGIS Online.)
Anyways, I've been looking into various resources for automating this process, such as connecting the SQL database via ArcGIS Pro (but I saw another thread on here mention the latency issues), the Esri Data Pipeline (but it doesn't automatically scrape data afaik), the Microsoft Data Pipeline (which I have no experience with), and Python. From what I've read, Python is the best option for scraping the website (or even from the SQL database). I was wondering if anyone has info on this, suggestions on what they use, or if there's any better options out there?
Leaning towards Python at the moment since I have coding experience with it. The issue I encounter is the 2FA authentication when entering the website, so I'd have to pull the code from my email, or I could set it up with the SQL database, or if anyone else has solutions to this. I don't have experience connecting to a database, so looking for any help I can get. Thank you so much!
