|
POST
|
Locks, by their very nature, tell you who has obtained them (host and connection user). This will tell you why they exist. The biggest offenders are feature dataset use (lock one, lock them all) and ArcGIS Server map services, but simply having ArcCatalog and ArcMap open at the same time can suffice (which is why I only use the Catalog window of ArcMap, not ArcCatalog). - V
... View more
04-01-2020
07:10 AM
|
0
|
15
|
7615
|
|
POST
|
Well, you could, but probably not through an Esri API. The "USE database" is incompatible with the Enterprise geodatabase connection file (.sde) being linked to a specific database. In theory, the USE isn't required, so an arcpy.ArcSDESQLExecute() request could issue the GRANT command, but the principal benefit of using an Enterprise geodatabase is to take advantage of versioning and geodatabase behaviors, and only a tool like ChangePrivileges is going to know which tables participate in geodatabase functionality, so if you roll your own, you might have permission issues later on down the road (for example, feature datasets have the permission of the least access of the members, so if you miss one GRANT, you might not have any access at all). - V
... View more
03-31-2020
09:49 PM
|
3
|
0
|
1164
|
|
BLOG
|
This blog rolls out version 2.0 of my jhuEmulator.py utility. The folks at Johns Hopkins University have done an awesome job at maintaining their Ops Dashboard site. They even share a folder of data snapshots as a CSV files at a minute before midnight, Greenwich time (2359 UTC). However, the data itself is anything but static, and anyone who needs the CSV files at a more frequent pitch, if not real-time, can be frustrated by the snapshot frequency. Fortunately, the data is all available with real-time updates -- It's in the feature service layers hosted on services1.arcgis.com (ncov_cases and ncov_cases_US). But there's one complication: The file format changed with the 2020-03-23.csv file, so anyone who had tools to read the old format was left in the lurch. I've written a Python utility which can: Export feature service data in either the old (Province/State,...) or new (FIPS,...) CSV format Export near real-time updates every 2 minutes, or as slowly as once every 12 hours Choose to skip export if the number of confirmed cases, deaths, or recoveries don't change in any particular time-slice (sometimes the Last_Update value changes, but the values reported don't) Attached to this blog post is a zipfile of the jhuEmulator.py utility. The usage looks like this: D:\covid-19>python jhuEmulator.py -h
usage: jhuEmulator.py [-h] [--adminLevel {0,1,2}] [--verbose VERBOSE]
[--interval INTERVAL] [--folder FOLDER]
[--csvFormat CSVFORMAT] [--stopPath STOPPATH]
[--skipTrivial SKIPTRIVIAL] [--usOnly USONLY]
[--confirmedOnly CONFIRMEDONLY]
[--topStates {5,10,15,20,25,30,35,40,45,all}]
{IMMEDIATE,DELAY,ONCE}
Emulate JHU COVID-19 data file (v2.0)
positional arguments:
{IMMEDIATE,DELAY,ONCE}
Execution mode
optional arguments:
-h, --help show this help message and exit
--adminLevel {0,1,2} (default = 2)
--verbose VERBOSE Verbose reporting flag (default = False)
--interval INTERVAL Data retrieval interval (default = 60m)
--folder FOLDER Folder path for data files (default = 'data')
--csvFormat CSVFORMAT
strftime format for data files
--stopPath STOPPATH File that indicates loop execution (default =
'stop.now')
--skipTrivial SKIPTRIVIAL
Defer writing insignificant changes flag (default =
False)
--usOnly USONLY Only export US data (default = False)
--confirmedOnly CONFIRMEDONLY
Only export rows with confirmed cases (default =
False)
--topStates {5,10,15,20,25,30,35,40,45,all}
Display sorted Confirmed/Deaths by US state (default =
0)
Can generate both CSV formats (before/after) 23-Mar-2020 The simplest use is a one-time execution (ONCE) mode: D:\covid-19>python jhuEmulator.py ONCE
1910: 2369 rows written ( 838061 / 41261 / 175737 )
D:\covid-19\demo>head data\2020-03-31_1910Z.csv
FIPS,Admin2,Province_State,Country_Region,Last_Update,Lat,Long_,Confirmed,Deaths,Recovered,Active,Combined_Key
45001,Abbeville,South Carolina,US,2020-03-31 18:31:52,34.223334,-82.461707,3,0,0,0,"Abbeville, South Carolina, US"
22001,Acadia,Louisiana,US,2020-03-31 18:31:52,30.295065,-92.414197,11,1,0,0,"Acadia, Louisiana, US"
51001,Accomack,Virginia,US,2020-03-31 18:31:52,37.767072,-75.632346,7,0,0,0,"Accomack, Virginia, US"
16001,Ada,Idaho,US,2020-03-31 18:31:52,43.452658,-116.241552,163,3,0,0,"Ada, Idaho, US"
19001,Adair,Iowa,US,2020-03-31 18:31:52,41.330756,-94.471059,1,0,0,0,"Adair, Iowa, US"
29001,Adair,Missouri,US,2020-03-31 18:31:52,40.190586,-92.600782,1,0,0,0,"Adair, Missouri, US"
40001,Adair,Oklahoma,US,2020-03-31 18:31:52,35.884942,-94.658593,4,0,0,0,"Adair, Oklahoma, US"
08001,Adams,Colorado,US,2020-03-31 18:31:52,39.874321,-104.336258,152,0,0,0,"Adams, Colorado, US"
17001,Adams,Illinois,US,2020-03-31 18:31:52,39.988156,-91.187868,2,0,0,0,"Adams, Illinois, US" The default is --adminLevel=2 (new-style), but levels 1 and 0 are also supported (zero is the same format as one, but without any state/province data for the US, Canada, China, or Australia): D:\covid-19>python jhuEmulator.py ONCE --adminLevel=1
1911: 315 rows written ( 838061 / 41261 / 175737 )
D:\covid-19>head data\2020-03-31_1910Z.csv
Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered,Latitude,Longitude
,Italy,2020-03-31T18:31:40,105792,12428,15729,41.871940,12.567380
,Spain,2020-03-31T18:31:40,94417,8269,19259,40.463667,-3.749220
New York,US,2020-03-31T18:31:52,75795,1550,0,42.165726,-74.948051
,Germany,2020-03-31T18:31:40,68180,682,15824,51.165691,10.451526
Hubei,China,2020-03-31T01:06:37,67801,3187,63153,30.975600,112.270700
,France,2020-03-31T18:31:40,52128,3523,9444,46.227600,2.213700
,Iran,2020-03-31T18:31:40,44605,2898,14656,32.427908,53.688046
,United Kingdom,2020-03-31T18:31:40,25150,1789,135,55.378100,-3.436000
New Jersey,US,2020-03-31T18:31:52,17126,198,0,40.298904,-74.521011
D:\covid-19>python jhuEmulator.py ONCE --adminLevel=0
1911: 180 rows written ( 838061 / 41261 / 175737 ) Note that successive executions in the same time window will overwrite the output file! The IMMEDIATE and DELAY options are nearly the same, except the IMMEDIATE makes a new snapshot without delay, while the DELAY mode only operates at regular intervals (all with a 1-20 second random delay, to prevent slamming the service with synchronized queries). The next two quote blocks were collected from two sessions running at the same time (in different directories): D:\Projects\covid-19>python jhuEmulator.py IMMEDIATE --interval=15m --adminLevel=1
1917: 315 rows written ( 838061 / 41261 / 175737 )
1930: 315 rows written ( 838061 / 41261 / 175737 )
1945: 315 rows written ( 838061 / 41261 / 175737 )
2000: 315 rows written ( 846156 / 41494 / 176171 )
2015: 315 rows written ( 846156 / 41494 / 176171 )
2030: 315 rows written ( 846156 / 41494 / 176171 )
2045: 315 rows written ( 846156 / 41494 / 176171 )
2100: 315 rows written ( 846156 / 41494 / 176171 )
2115: 315 rows written ( 846156 / 41494 / 176171 )
2130: 315 rows written ( 850583 / 41654 / 176714 )
2145: 315 rows written ( 850583 / 41654 / 176714 )
2200: 315 rows written ( 850583 / 41654 / 176714 )
2215: 315 rows written ( 850583 / 41654 / 176714 )
2230: 315 rows written ( 855007 / 42032 / 177857 ) D:\covid-19>python jhuEmulator.py DELAY --interval=15m --adminLevel=1 --skipTrivial=True
1930: 315 rows written ( 838061 / 41261 / 175737 )
2000: 315 rows written ( 846156 / 41494 / 176171 )
2130: 315 rows written ( 850583 / 41654 / 176714 )
2230: 315 rows written ( 855007 / 42032 / 177857 ) Note that the --skipTrivial=True flag is the mechanism to skip data export if no significant changes have occurred (changes to the Last_Updated field will not write a new file, but changes to any case count field will). Assembling the new-style data format is tricky, because it needs to aggregate thousands of records from two different services (and remove the US duplicates from the ncov_cases service), the verbose mode gives an indication of what's happening: D:\covid-19>python jhuEmulator.py ONCE --verbose=True
Querying 'ncov_cases_US' service (1/5)...
382.2 KB retrieved (476ms elapsed)
Querying 'ncov_cases_US' service (2/5)...
281.9 KB retrieved (250ms elapsed)
Querying 'ncov_cases_US' service (3/5)...
200.1 KB retrieved (176ms elapsed)
Querying 'ncov_cases_US' service (4/5)...
262.2 KB retrieved (204ms elapsed)
Querying 'ncov_cases_US' service (5/5)...
169.5 KB retrieved (161ms elapsed)
Querying 'ncov_cases' service...
163.6 KB retrieved (169ms elapsed)
Creating datafile '2020-03-31_1936Z.csv'...
2369 rows written ( 838061 / 41261 / 175737 ) For the US-centric audience, I added a --topStates flag (which must be used with --verbose=Y) to print a summary D:\covid-19>python jhuEmulator.py ONCE --verbose=True --topStates=15
Querying 'ncov_cases_US' service (1/5)...
390.0 KB retrieved (429ms elapsed)
Querying 'ncov_cases_US' service (2/5)...
286.2 KB retrieved (168ms elapsed)
Querying 'ncov_cases_US' service (3/5)...
200.7 KB retrieved (189ms elapsed)
Querying 'ncov_cases_US' service (4/5)...
268.2 KB retrieved (163ms elapsed)
Querying 'ncov_cases_US' service (5/5)...
172.5 KB retrieved (157ms elapsed)
===== Top 15 States =====
State Confirmed Deaths Counties
New York 75798 1550 56
New Jersey 18696 267 22
California 8077 163 48
Michigan 7615 259 69
Massachusetts 6620 89 14
Florida 6338 77 54
Illinois 5994 99 54
Washington 5305 222 35
Louisiana 5237 239 61
Pennsylvania 4963 63 60
Georgia 3815 111 138
Texas 3726 53 131
Connecticut 3128 69 9
Colorado 2627 51 48
Tennessee 2391 23 82
*Others 25475 462 1247
Querying 'ncov_cases' service...
163.6 KB retrieved (132ms elapsed)
Creating datafile '2020-03-31_2220Z.csv'...
2405 rows written ( 855007 / 42032 / 177857 )
D:\covid-19>python jhuEmulator.py ONCE --verbose=True --topStates=15 --adminLevel=1
Querying 'ncov_cases' service...
163.6 KB retrieved (442ms elapsed)
===== Top 15 States =====
State Confirmed Deaths
New York 75798 1550
New Jersey 18696 267
California 8077 163
Michigan 7615 259
Massachusetts 6620 89
Florida 6338 77
Illinois 5994 99
Washington 5305 222
Louisiana 5237 239
Pennsylvania 4963 63
Georgia 3815 111
Texas 3726 53
Connecticut 3128 69
Colorado 2627 51
Tennessee 2391 23
*Others 25475 462
Creating datafile '2020-03-31_2221Z.csv'...
315 rows written ( 855007 / 42032 / 177857 ) The format difference is due to the fact that Admin2 reporting is by county, but doesn't populate Recovered and Active, while the Admin1 reporting includes Recovered (which would allow Active to be computed, but Recovered is now a lump-sum record in the ncov_cases service, so in practice, Active and Recovered are not available). So, how do I know this script populates the same data shared on GitHub? Well, I wrote a validator utility to create two CovidSummary objects, then iterate from one searching the other for duplicate keys, and reporting missing rows, data mismatches, and unmatched rows, and the output was: D:\covid-19>python validate.py
========================================================================
0 errors / 3429 lines Okay, I cheated a bit, since the daily JHU CSV reports that the Diamond Princess and Grand Princess cruise ships are docked on Null Island, I treated 0.0 degrees latitude/longitude as a wildcard that matches any coordinate, and I only compare coordinates to 5 places, since string comparison doesn't work well with floating-point values, but everything else has aligned perfectly. UPDATE @ 2100 EST: Unfortunately, sometime today, the US Admin2 jurisdictions without any confirmed cases were deleted from the ncov_cases_US service feed, the FIPS code displayed for the Northern Mariana Islands and US Virgin Islands disappeared, and the US territory without any confirmed cases (American Samoa) also disappeared (along with its FIPS code). I've tweaked the exporter to conform to this, and it runs cleanly now: D:\covid-19>python validate.py
========================================
sum1 = data2\2020-03-31.csv
sum2 = data2\2020-04-01_0110Z.csv
0 errors / 2434 lines But I'm expecting Guam to have its FIPS code removed in the coming days. Since the validation tool implements Python classes to parse and search both old-style and new-style CSV files, I've attached that as well (as validator-v10.zip) My next task is to write some code to exploit this near real-time data resource, and use it to maintain PostgreSQL tables to produce clones of the ncov_cases* services' data. Now attached is jhuEmulator-v20.zip -- Changelog: Implemented adminLevel=2 with multiple queries to replicate three-level results in GitHub CSV files Added --confirmedOnly flag to skip counties without Confirmed cases (obviated by feed change) Added --topStates to report ordered state-wide impact (requires --verbose=True and non-zero --adminLevel) Added FIPS lookup for US Possessions (Note: Puerto Rico = None) Ignored US admin1 summary values in admin2 feed (2-digit FIPS) Added validateKey to correct Combined_Key with missing spaces after commas Removed FIPS from US possessions and suppressed US possessions without confirmed cases (as per 2020-03-31.csv) == Update 01-Apr @ 0100 EST == Long-running service execution is great for finding bugs... Now attached is jhuEmulator-v21.zip -- Changelog: Fixed significantChange field comparison error with Admin1 field names in Admin2 service == Update 01-Apr @ 0920 EST == Overnight service execution is great for finding bugs, but errors that arise after several hours are less fun... Now attached is jhuEmulator-v22.zip -- Changelog: Fixed significantChange field comparison error for *all* Admin1 field names in the Admin2 service Fixed paren alignment defect in error handling in pullData Also attached is covidValidate_v11.zip -- Changelog: Tightened CovidReport.compare() loop Fixed range check defect in main == Update 01-Apr @ 2030 EST == The server glitched, dropping all US data, and I got to exercise some error code that doesn't normally see traffic, which I've now tweaked to be more resilient. Attached is jhuEmulator-v23.zip -- Changelog: Fixed TypeError when service result is empty (None) == Update 08-Apr @ 1600 EST == I tweaked the exporter to always write data at UTC midnight, so that the validator has the same Last_Update value for comparison with the JHU published file. I had to improve date parsing in the CovidReport class (the corrected 2020-04-06.csv has a different date format). I'm still seeing some JHU data with incorrect 000xx FIPS codes, and the Admin1 and Admin2 feeds are out of sync for the Northern Mariana Islands, but otherwise looking good. Attached is jhuEmulator-v24.zip -- Changelog: Tweak to force 0000Z download, even when --skipTrivial is enabled (for comparison with GitHub published files) Also attached is covidValidate_v12.zip -- Changelog: Cleaned up CovidReport.extract() to handle bad formatting gracefully and to support the 'm/d/y HH:MM' date used in the revised 2020-04-06.csv datafile Loosened the Last_Update comparison code so that second- truncated timestamps can compare successfully == Update 12-Apr @ 2320 EST == The JHU daily CSV format changed again, adding five more fields (and moving FIPS later in the display order). I've got the covidValidate.py handling the new format, but am not yet generating it... Attached is covidValidate_v30.zip -- Changelog: Count, but don't display, Combined_Key mismatches Added parsing (but not yet validation) of fields added to 04-12-2020.csv - V
... View more
03-31-2020
06:20 PM
|
3
|
5
|
4743
|
|
POST
|
I regularly load 2M, 3M, 5M, 8M, and 12M row tables into PostgreSQL, and occasionally go as high as 60M rows. I use both FeatureClassToFeatureClass / TableToTable (as appropriate) or a nested arcpy.da.SearchCursor / arcpy.da.InsertCursor pair. Performance is usually about the same, taking 20-240 minutes, depending on size of the data and the target database (effectively the same rate for each DB). I have seen multi-day loads, but only when trying something sub-optimal, like loading into an AWS RDS across three routers shared by hundreds of users. My best performance has occurred on a high-IOPS EC2 in the same mission as the high-IOPS RDS, using ArcPy from the Linux EC2 AGS host, where a cascade of FC2FC commands (40 some-odd tables) populated 35-40M rows in ~2 hours. I have cheated on occasion, by parallelizing my load scripts, so one PuTTY session was loading the odd tables while the other PuTTY session was loading the even ones (with load order sequenced by size, so they finished about the same time). You'd need to provide a great deal more information about the network architecture, the capabilities of the loading and database host, and location of the source FGDB before specific advice could be offered. - V
... View more
03-24-2020
06:14 PM
|
2
|
0
|
2555
|
|
BLOG
|
Like many, I've been using the daily data from the Johns Hopkins University Ops Dashboard site, but I've been frustrated by the periodicity and timing -- Midnight UTC is all well and good, but then you've got 12+ hour old data before folks start their workday on the US East Coast. But there is another route to the data, since it's published as a feature service. So Sunday night I whipped up a Python script to pull the full feature feed from AGOL, and write an hourly file in the same format. This script is plain-vanilla Python (no arcpy), and has been tested on both Python 2.7.16 and 3.8.2. Execution looks looks something like this: D:\covid-19>python jhuEmulator.py DELAY --verbose=True --interval=2h
Sleeping 35.67 minutes...
Querying feature service...
155.6 KB retrieved (310ms elapsed)
Creating datafile '2020-03-25_0600Z.csv'...
300 rows written ( 422989 / 18916 / 108578 )
Sleeping 2.00 hours...
Querying feature service...
155.6 KB retrieved (179ms elapsed)
Creating datafile '2020-03-25_0800Z.csv'...
300 rows written ( 423670 / 18923 / 108629 )
Sleeping 2.00 hours...
Querying feature service...
155.6 KB retrieved (180ms elapsed)
Creating datafile '2020-03-25_1000Z.csv'...
300 rows written ( 425493 / 18963 / 109191 )
Sleeping 2.00 hours...
Querying feature service...
156.1 KB retrieved (146ms elapsed)
Creating datafile '2020-03-25_1200Z.csv'...
301 rows written ( 435006 / 19625 / 111822 )
Sleeping 2.00 hours...
Exiting on 'stop.now' request
D:\covid-19>dir/b data
2020-03-25_0600Z.csv
2020-03-25_0800Z.csv
2020-03-25_1000Z.csv
2020-03-25_1200Z.csv
D:\covid-19>head data\2020-03-25_1200Z.csv
Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered,Latitude,Longitude
,Italy,2020-03-25T07:24:40,69176,6820,8326,41.87194,12.56738
Hubei,China,2020-03-25T02:43:02,67801,3163,60810,30.9756,112.2707
,Spain,2020-03-25T07:24:40,47610,3434,5367,40.463667,-3.74922
,Germany,2020-03-25T07:24:40,34009,172,3532,51.165691,10.451526
,Iran,2020-03-25T07:24:40,27017,2077,9625,32.427908,53.688046
New York,US,2020-03-25T07:24:54,26376,271,0,42.165726,-74.948051
,France,2020-03-25T07:24:40,22304,1100,3281,46.2276,2.2137
,Switzerland,2020-03-25T07:24:40,10171,135,131,46.8182,8.2275
,Korea, South,2020-03-25T07:24:40,9137,126,3730,35.907757,127.766922 Then I worried about system load (One ought not pound the free service that one wants to use into oblivion) and update frequency (Wouldn't it be nice to get updates as they happen?), so I started working on tweaks: Added argparse command-line switches to change execution parameters (vice hacking the source) Added 0-20 random seconds delay for execution of the query (so synchronized servers don't overload the feature service, yet with plenty of time to finish within the targeted minute) Permitted a directory tree archiving style (creating directories if necessary) I started working on a way to only write a dataset which has had a substantive change over the previous timestamp (so the timing could be shortened to, say, 5 minutes, without needlessly transmitting files), but I discovered I can't use the Last_Update field for this (since the timestamp changes without case count changes), and then the CSV data format posted on GitHub changed Monday night to include Admin2 (county-level) data in the US (at lower resolution -- no Recovered/Active values), so I'll post this as-is for reference (or for those that now need an alternate source for data in the old format) and figure out how to model the new format... Note that there are three modes (you need to choose one): ONCE - Create one file, now, and exit (suitable for a cron job or other batch-oriented invocation at an interval greater than one hour) IMMEDIATE - Create one file now, and then continue file creation based on the interval modulus (default hourly) DELAY - Create first file in next expected time window (if the old job was killed, and you just want to pick up where left off) It's possible to place files in a directory tree based on the time format. For example, if you specify --csvFormat="y%Y\m%m\d%d\t%H%MZ.csv" Then the yYYYY\mMM\dDD folder will be created if necessary, and the "tHHMMZ.csv" data file added to that. You can gracefully exit the application by creating a "stop.now" file in the active directory (or creating the file referenced by the optional --stopPath flag). == Update 25-Mar @ 0800 EST == Now attached is jhuEmulator-v11.zip -- Changelog: Parameterized URL assembly for future US feed support Added mapping for missing fields (to None value) Fixed stopPath utilization defect Fixed sys.exit() defect Reduced wait for stopPath detection Expanded interval to even divisions of hour and day (2m,3m,4m,5m,6m,10m,12m,15m,30m,1h,2h,3h,4h,6h,12h) I should probably re-emphasize that this tool can be used to preserve the "old format" feed until your tools have the ability to process the new format with FIPS and Admin2 fields. == Update 25-Mar @ 0840 EST == Now attached is jhuEmulator-v12.zip -- Changelog: Fixed IMMEDIATE computeDelay() defect Note that the stop file will now (as of v11) gracefully exit the app within a minute of creation. == Update 25-Mar @ 1520 EST == Now attached is jhuEmulator-v13.zip -- Changelog: Detect low-volume data pulls and skip write operation Fixed defect in byVolume sorting (for/else) Fixed defect in pull error logic 0.4 millisecond/existence test timing compensation Made Interrupt handling in sleep cleaner Working on nominal change detection and US counties feed support next... == Update 25-Mar @ 1820 EST == Now attached is jhuEmulator-v14.zip -- Changelog: Added substantive change detection (only write if case counts or Lat/Lon are different, not if just Last_Update changed) - Uses new --skipTrivial flag (default False) Fixed naming defect in main loop Self-initializing record count expectation The --skipTrivial flag is what I had been looking to achieve in near-realtime feed support. With it, you can set a short interval (though not less than 2 minutes) and have CSV files only written when "substantive change" is detected (row count or changes to Confirmed, Deaths, or Reported field values, mostly). Executing it with --interval=2m (which is probably way too often -- 5, 6, 10, 12, or 15 would be better) results in output like this: D:\covid-19>python jhuEmulator.py IMMEDIATE ^
More? --verbose=True --skipTrivial=True --interval=2m
Querying feature service...
156.6 KB retrieved (232ms elapsed)
Creating datafile '2020-03-25_2207Z.csv'...
302 rows written ( 464026 / 20946 / 113691 )
Sleeping 0.15 minutes...
Querying feature service...
156.6 KB retrieved (152ms elapsed)
Skipping write of trivial change to '2020-03-25_2208Z.csv'...
Sleeping 2.07 minutes...
Querying feature service...
156.6 KB retrieved (195ms elapsed)
Skipping write of trivial change to '2020-03-25_2210Z.csv'...
Sleeping 2.16 minutes...
Querying feature service...
156.6 KB retrieved (176ms elapsed)
Skipping write of trivial change to '2020-03-25_2212Z.csv'...
Sleeping 1.80 minutes...
Querying feature service...
156.6 KB retrieved (187ms elapsed)
Skipping write of trivial change to '2020-03-25_2214Z.csv'...
Sleeping 2.20 minutes...
Querying feature service...
157.2 KB retrieved (185ms elapsed)
Creating datafile '2020-03-25_2216Z.csv'...
303 rows written ( 466836 / 21152 / 113769 )
Sleeping 1.88 minutes...
Querying feature service...
157.2 KB retrieved (179ms elapsed)
Skipping write of trivial change to '2020-03-25_2218Z.csv'... My next goal is to write data in the new (post 23-Mar) JHU CSV format from the ..._US service. I'll likely set it up to be both global and US-only. Please let me know if you use this or encounter problems... == Update 25-Mar @ 2200 EST == Blog title updated. == Update 26-Mar @ 1800 EST == Now attached is jhuEmulator-v15.zip -- Changelog: Added time prefix to non-verbose output Added --adminLevel=[0,1] flag to permit country summaries Added --usOnly support to restrict results to US data Fixed ONCE defect when retrieval fails Fixed CSV quoting bug with Python 3 Note: adminLevel=2 is partially functioning, but disabled until a work-around for the 1000 feature limit is added Good news: jhuEmulator has code to export new-style format files Bad news: Only 1000 rows can be received per query I did add the --adminLevel flag, but right now it only functions to summarize the countries that are usually distributed with province/state level data (US, Canada, China, and Australia). Python 3 users: Note that v1.5 fixes a formatting defect that failed to place double-quotes around values with embedded quotes. The next snapshot should be v2.0, which will assemble multiple queries into a single file which approximates the new-format CSV style. == Update 29-Mar @ 2330 EST == Now attached is jhuEmulator-v16.zip -- Changelog: Fixed timezone defect in datetime.datetime.fromtimestamp call (Last_Update had been captured in local timezone, not UTC) Added date display in non-verbose operation at midnight UTC I'm still working on the Admin2 format support. It's nearly done, but not ready for release. I discovered an ugly bug in the v15 and earlier extraaction -- the UTC dates in Last_Update are recorded in the local timezone, not UTC, so I wanted to get this fix out there, BUT I'm now showing far more cases than the JHU Ops Dashboard, so the code that was supposed to accumulate admin2 data into an admin1 summary seems to be double-counting. The same issue exists in v15 and earlier (and v20, right now), so I'll need to figure out what went wrong... == Update 29-Mar @ 2330 EST == Heh. My Chrome session needed refresh. The total in the website correlates to the extraction feed. For what it's worth, here's the log of my last 26 hours of data extraction, pulling data every 15 minutes, but only saving if if there was non-trivial change: D:\covid-19>python jhuEmulator.py DELAY --interval=15m --skipTrivial=True
0245: 312 rows written ( 663828 / 30822 / 139451 )
0300: 312 rows written ( 664608 / 30846 / 140156 )
0400: 312 rows written ( 664695 / 30847 / 140156 )
0515: 312 rows written ( 664924 / 30848 / 140222 )
0630: 312 rows written ( 665164 / 30852 / 140225 )
0745: 312 rows written ( 665616 / 30857 / 141746 )
0845: 312 rows written ( 666211 / 30864 / 141789 )
0945: 312 rows written ( 669312 / 30982 / 142100 )
1100: 312 rows written ( 678720 / 31700 / 145609 )
1200: 312 rows written ( 679977 / 31734 / 145625 )
1315: 312 rows written ( 681706 / 31882 / 145696 )
1430: 312 rows written ( 684652 / 32113 / 145696 )
1530: 312 rows written ( 685623 / 32137 / 145706 )
1645: 312 rows written ( 691867 / 32988 / 146613 )
1815: 312 rows written ( 704095 / 33509 / 148824 )
1900: 312 rows written ( 710918 / 33551 / 148900 )
2000: 312 rows written ( 713171 / 33597 / 148995 )
2130: 312 rows written ( 716101 / 33854 / 149071 )
2230: 312 rows written ( 718685 / 33881 / 149076 )
2345: 312 rows written ( 720117 / 33925 / 149082 )
----> Mon 2020-03-30 UTC
0045: 312 rows written ( 721584 / 33958 / 149122 )
0200: 312 rows written ( 721817 / 33968 / 151204 )
0300: 312 rows written ( 722289 / 33984 / 151901 )
0400: 312 rows written ( 722435 / 33997 / 151991 ) == Update 31-Mar @ 2120 EST == The Admin2 format emulator jhuEmulator-v20.zip is now published in the new blog post: https://community.esri.com/people/vangelo-esristaff/blog/2020/04/01/python-utility-to-write-covid-19-csv-snapshots-from-arcgiscom-feature-services-v2 There were output changes only today, so v2.0 is likely to to have some updates to keep in sync. The most significan change for anyone using the v1.x tools is that the new default file format is the post-03/23 format, but that can be overridden with --adminVersion=1. - V
... View more
03-24-2020
01:40 PM
|
1
|
0
|
1449
|
|
POST
|
Cheryl reported the bug number - BUG-000117499 - which does not appear to have any updates. I don't have anything to add here. You might want to try Tech Support, adding your e-mail to the notifications on this issue... - V
... View more
02-14-2020
08:37 AM
|
1
|
0
|
3498
|
|
POST
|
Cross-posted to gis.stackexchange.com as arcgis 10.7 - Cannot delete Feature Class, after having un-registered as versioned. The item does not have a definition …
... View more
12-02-2019
09:31 AM
|
1
|
0
|
1328
|
|
POST
|
Actually, you are not asking how the technology works, but how it is licensed, which is a legal question. Since an incorrect answer provided by anyone except an official Esri representative does not protect you from a lawsuit, you probably ought to be more concerned about getting official Esri clarification (which is not generally provided in this community). If you had consulted with your own legal staff, and the documentation I referenced did not make it clear, they would want you to have something in writing, or at least in a one-on-one discussion with resellers, with a follow-up e-mail. I did not, and will not, explicitly answer the question because my answer has no value. If all the documentation references leave you unsure of their meaning, then you should contact your legal staff or Esri's formal representatives for clarification. In the end, free legal advice on the Internet isn't worth what you paid for it. - V
... View more
11-29-2019
09:02 AM
|
3
|
1
|
5460
|
|
POST
|
So here's what the the documentation says (my emphasis): The ArcGIS Server component of ArcGIS Enterprise is licensed by CPU cores. Cores are part of the computer’s CPU (Central Processing Unit) that determines the number of instructions that can be processed at a given time. A 4-core processor can handle four sets of instructions at once. Many modern CPUs further support a technology known as hyper-threading that allows a single core to run multiple concurrent sets of instructions. Hyper-threads do not count towards the licensed cores. Licensed cores can be counted by physical cores or virtual cores, whichever is more advantageous to you. The minimum purchase for ArcGIS Enterprise is an “up-to-4 core” license* which can be installed on a two or four core machine. If your machine has more than 4 cores (physical or virtual), you can purchase additional cores, or virtualize the machine so that the 4-core license of ArcGIS Enterprise is only installed on 4 (virtual) cores and thereby satisfies the license requirement. If a 4-core license is installed on an 8-core physical machine without virtualization, then ArcGIS Enterprise will use all 8 cores and would be in violation of the license agreement until you purchase a license for the four “additional cores”. And further: Note that early generations of Microsoft Azure VMs used physical core equivalents, listed in terms of “cores”, whereas later generation Microsoft Azure VMs use vCPUs that correspond to a hyper-thread. It is important to calculate the physical core equivalent correctly based on the specific generation of VM that is chosen. *Note: ArcGIS Enterprise requires a minimum initial license purchase of 4 cores per physical location. The minimum for ArcGIS Enterprise Workgroup is 2 cores. Additional cores may be licensed in 2-core increments. ArcGIS Enterprise Workgroup is not licensed for multi-machine environments If you find this unclear, you should contact Customer Service, Tech Support, or your local marketing representative. - V
... View more
11-26-2019
10:34 AM
|
4
|
6
|
19193
|
|
POST
|
Directly modifying rows in the SDE-owned tables outside of the stored procedures invoked by ArcObjects is very unsupported. Even if the existing flags were documented, the only supported use would be for read-only metadata purposes. All other uses are likely to result in geodatabase corruption (in the post-9.x denormalized implementation, metadata is redundantly stored for performance optimization; if you miss even one location, integrity is compromised, which could cause upgrade failure years later). I suggest you contact Tech Support for a less risky procedure. - V
... View more
11-13-2019
07:47 AM
|
0
|
0
|
2728
|
|
POST
|
The '.gdb' of a file geodatabase is a directory, not a file. The file geodatabase is the directory and all the files below it (except the lock files -- you can keep those out). - V
... View more
10-25-2019
11:13 AM
|
2
|
1
|
12774
|
|
POST
|
Network drives are bad juju for file geodatabase. Always use a local drive for I/O-intensive operations. - V
... View more
09-17-2019
01:05 PM
|
1
|
1
|
5436
|
|
POST
|
It's a violation of best practice to load spatial data as the 'sde' login, much the way you shouldn't do software development as 'root' or Administrator. Instead, create data management logins, schemas that match, and load data as that login. For example, if I have streets data managed by the Pavement department, I'll create a login called "pavement" and a schema "pavement" then load the data in the production database as "prod.pavement.streets". Best practice also calls for role creation, with access rights granted by role to the feature classes. Then normal logins associated with individual department staff members can be granted appropriate roles by resource (GRANT pavement_editor TO bob_smith). It's also best practice to avoid using data owners in web application connections, so a "viewer_app" login/schema could be created, with an appropriate role (GRANT pavement_viewer TO viewer_app). - V
... View more
06-12-2019
06:27 AM
|
0
|
1
|
2007
|
|
POST
|
Unfortunately, the ArcSDE API was deprecated at 10.2, and hasn't been available since 10.3. `se_toolkit` survived (for 10.2.2 and before) until the FTP server was decommissioned. You can certainly execute arbitrary SQL from ArcPy (with arcpy.ArcSDESQLExecute) and INSERT with arcpy.da.InsertCursor and UPDATE and DELETE with arcpy.da.UpdateCursor. - V
... View more
05-06-2019
08:20 AM
|
0
|
0
|
1582
|
|
POST
|
The Server product was also called "ArcGIS Server" before being called "ArcGIS for Server" (which I always thought silly), before being named "ArcGIS Server" again, and then bundled as a component in "ArcGIS Enterprise". As Romeo well knew, the name does not alter the flower to which it refers (to extend that metaphor, Enterprise is a garden). - V
... View more
05-03-2019
05:18 PM
|
1
|
0
|
1288
|
| Title | Kudos | Posted |
|---|---|---|
| 1 | Tuesday | |
| 2 | 3 weeks ago | |
| 1 | 05-29-2026 12:51 PM | |
| 1 | a month ago | |
| 2 | 05-29-2026 08:31 AM |