Latest Contributions by VinceAngelo

This blog rolls out version 2.0 of my jhuEmulator.py utility. The folks at Johns Hopkins University have done an awesome job at maintaining their Ops Dashboard site.  They even share a folder of data snapshots as a CSV files at a minute before midnight, Greenwich time (2359 UTC).  However, the data itself is anything but static, and anyone who needs the CSV files at a more frequent pitch, if not real-time, can be frustrated by the snapshot frequency. Fortunately, the data is all available with real-time updates -- It's in the feature service layers hosted on services1.arcgis.com (ncov_cases and ncov_cases_US). But there's one complication: The file format changed with the 2020-03-23.csv file, so anyone who had tools to read the old format was left in the lurch. I've written a Python utility which can: Export feature service data in either the old (Province/State,...) or new (FIPS,...) CSV format Export near real-time updates every 2 minutes, or as slowly as once every 12 hours Choose to skip export if the number of confirmed cases, deaths, or recoveries don't change in any particular time-slice (sometimes the Last_Update value changes, but the values reported don't) Attached to this blog post is a zipfile of the jhuEmulator.py utility.  The usage looks like this: D:\covid-19>python jhuEmulator.py -h usage: jhuEmulator.py [-h] [--adminLevel {0,1,2}] [--verbose VERBOSE]                       [--interval INTERVAL] [--folder FOLDER]                       [--csvFormat CSVFORMAT] [--stopPath STOPPATH]                       [--skipTrivial SKIPTRIVIAL] [--usOnly USONLY]                       [--confirmedOnly CONFIRMEDONLY]                       [--topStates {5,10,15,20,25,30,35,40,45,all}]                       {IMMEDIATE,DELAY,ONCE} Emulate JHU COVID-19 data file (v2.0) positional arguments:   {IMMEDIATE,DELAY,ONCE}                         Execution mode optional arguments:   -h, --help            show this help message and exit   --adminLevel {0,1,2}  (default = 2)   --verbose VERBOSE     Verbose reporting flag (default = False)   --interval INTERVAL   Data retrieval interval (default = 60m)   --folder FOLDER       Folder path for data files (default = 'data')   --csvFormat CSVFORMAT                         strftime format for data files   --stopPath STOPPATH   File that indicates loop execution (default =                         'stop.now')   --skipTrivial SKIPTRIVIAL                         Defer writing insignificant changes flag (default =                         False)   --usOnly USONLY       Only export US data (default = False)   --confirmedOnly CONFIRMEDONLY                         Only export rows with confirmed cases (default =                         False)   --topStates {5,10,15,20,25,30,35,40,45,all}                         Display sorted Confirmed/Deaths by US state (default =                         0) Can generate both CSV formats (before/after) 23-Mar-2020‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The simplest use is a one-time execution (ONCE) mode: D:\covid-19>python jhuEmulator.py ONCE   1910:   2369 rows written (  838061 /  41261 /  175737 ) D:\covid-19\demo>head data\2020-03-31_1910Z.csv FIPS,Admin2,Province_State,Country_Region,Last_Update,Lat,Long_,Confirmed,Deaths,Recovered,Active,Combined_Key 45001,Abbeville,South Carolina,US,2020-03-31 18:31:52,34.223334,-82.461707,3,0,0,0,"Abbeville, South Carolina, US" 22001,Acadia,Louisiana,US,2020-03-31 18:31:52,30.295065,-92.414197,11,1,0,0,"Acadia, Louisiana, US" 51001,Accomack,Virginia,US,2020-03-31 18:31:52,37.767072,-75.632346,7,0,0,0,"Accomack, Virginia, US" 16001,Ada,Idaho,US,2020-03-31 18:31:52,43.452658,-116.241552,163,3,0,0,"Ada, Idaho, US" 19001,Adair,Iowa,US,2020-03-31 18:31:52,41.330756,-94.471059,1,0,0,0,"Adair, Iowa, US" 29001,Adair,Missouri,US,2020-03-31 18:31:52,40.190586,-92.600782,1,0,0,0,"Adair, Missouri, US" 40001,Adair,Oklahoma,US,2020-03-31 18:31:52,35.884942,-94.658593,4,0,0,0,"Adair, Oklahoma, US" 08001,Adams,Colorado,US,2020-03-31 18:31:52,39.874321,-104.336258,152,0,0,0,"Adams, Colorado, US" 17001,Adams,Illinois,US,2020-03-31 18:31:52,39.988156,-91.187868,2,0,0,0,"Adams, Illinois, US"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The default is --adminLevel=2 (new-style), but levels 1 and 0 are also supported (zero is the same format as one, but without any state/province data for the US, Canada, China, or Australia): D:\covid-19>python jhuEmulator.py ONCE --adminLevel=1   1911:    315 rows written (  838061 /  41261 /  175737 ) D:\covid-19>head data\2020-03-31_1910Z.csv Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered,Latitude,Longitude ,Italy,2020-03-31T18:31:40,105792,12428,15729,41.871940,12.567380 ,Spain,2020-03-31T18:31:40,94417,8269,19259,40.463667,-3.749220 New York,US,2020-03-31T18:31:52,75795,1550,0,42.165726,-74.948051 ,Germany,2020-03-31T18:31:40,68180,682,15824,51.165691,10.451526 Hubei,China,2020-03-31T01:06:37,67801,3187,63153,30.975600,112.270700 ,France,2020-03-31T18:31:40,52128,3523,9444,46.227600,2.213700 ,Iran,2020-03-31T18:31:40,44605,2898,14656,32.427908,53.688046 ,United Kingdom,2020-03-31T18:31:40,25150,1789,135,55.378100,-3.436000 New Jersey,US,2020-03-31T18:31:52,17126,198,0,40.298904,-74.521011 D:\covid-19>python jhuEmulator.py ONCE --adminLevel=0   1911:    180 rows written (  838061 /  41261 /  175737 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Note that successive executions in the same time window will overwrite the output file! The IMMEDIATE and DELAY options are nearly the same, except the IMMEDIATE makes a new snapshot without delay, while the DELAY mode only operates at regular intervals (all with a 1-20 second random delay, to prevent  slamming the service with synchronized queries).  The next two quote blocks were collected from two sessions running at the same time (in different directories):   D:\Projects\covid-19>python jhuEmulator.py IMMEDIATE --interval=15m --adminLevel=1   1917:    315 rows written (  838061 /  41261 /  175737 )‍‍ ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍  1930:    315 rows written (  838061 /  41261 /  175737 )   1945:    315 rows written (  838061 /  41261 /  175737 )   2000:    315 rows written (  846156 /  41494 /  176171 )   2015:    315 rows written (  846156 /  41494 /  176171 )   2030:    315 rows written (  846156 /  41494 /  176171 )   2045:    315 rows written (  846156 /  41494 /  176171 )   2100:    315 rows written (  846156 /  41494 /  176171 )   2115:    315 rows written (  846156 /  41494 /  176171 )   2130:    315 rows written (  850583 /  41654 /  176714 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍   2145:    315 rows written (  850583 /  41654 /  176714 )   2200:    315 rows written (  850583 /  41654 /  176714 )   2215:    315 rows written (  850583 /  41654 /  176714 )   2230:    315 rows written (  855007 /  42032 /  177857 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ D:\covid-19>python jhuEmulator.py DELAY --interval=15m --adminLevel=1 --skipTrivial=True‍   1930:    315 rows written (  838061 /  41261 /  175737 )   2000:    315 rows written (  846156 /  41494 /  176171 )   2130:    315 rows written (  850583 /  41654 /  176714 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍   2230:    315 rows written (  855007 /  42032 /  177857 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Note that the --skipTrivial=True flag is the mechanism to skip data export if no significant changes have occurred (changes to the Last_Updated field will not write a new file, but changes to any case count field will). Assembling the new-style data format is tricky, because it needs to aggregate thousands of records from two different services (and remove the US duplicates from the ncov_cases service), the verbose mode gives an indication of what's happening: D:\covid-19>python jhuEmulator.py ONCE --verbose=True Querying 'ncov_cases_US' service (1/5)...    382.2 KB retrieved (476ms elapsed) Querying 'ncov_cases_US' service (2/5)...    281.9 KB retrieved (250ms elapsed) Querying 'ncov_cases_US' service (3/5)...    200.1 KB retrieved (176ms elapsed) Querying 'ncov_cases_US' service (4/5)...    262.2 KB retrieved (204ms elapsed) Querying 'ncov_cases_US' service (5/5)...    169.5 KB retrieved (161ms elapsed) Querying 'ncov_cases' service...    163.6 KB retrieved (169ms elapsed) Creating datafile '2020-03-31_1936Z.csv'...      2369 rows written (  838061 /  41261 /  175737 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ For the US-centric audience, I added a --topStates flag (which must be used with --verbose=Y) to print a summary  D:\covid-19>python jhuEmulator.py ONCE --verbose=True --topStates=15 Querying 'ncov_cases_US' service (1/5)...    390.0 KB retrieved (429ms elapsed) Querying 'ncov_cases_US' service (2/5)...    286.2 KB retrieved (168ms elapsed) Querying 'ncov_cases_US' service (3/5)...    200.7 KB retrieved (189ms elapsed) Querying 'ncov_cases_US' service (4/5)...    268.2 KB retrieved (163ms elapsed) Querying 'ncov_cases_US' service (5/5)...    172.5 KB retrieved (157ms elapsed)                       ===== Top 15 States =====                 State     Confirmed   Deaths  Counties                 New York     75798     1550        56               New Jersey     18696      267        22               California      8077      163        48                 Michigan      7615      259        69            Massachusetts      6620       89        14                  Florida      6338       77        54                 Illinois      5994       99        54               Washington      5305      222        35                Louisiana      5237      239        61             Pennsylvania      4963       63        60                  Georgia      3815      111       138                    Texas      3726       53       131              Connecticut      3128       69         9                 Colorado      2627       51        48                Tennessee      2391       23        82                  *Others     25475      462      1247 Querying 'ncov_cases' service...    163.6 KB retrieved (132ms elapsed) Creating datafile '2020-03-31_2220Z.csv'...      2405 rows written (  855007 /  42032 /  177857 ) D:\covid-19>python jhuEmulator.py ONCE --verbose=True --topStates=15 --adminLevel=1 Querying 'ncov_cases' service...    163.6 KB retrieved (442ms elapsed)                  ===== Top 15 States =====                 State     Confirmed   Deaths                 New York     75798     1550               New Jersey     18696      267               California      8077      163                 Michigan      7615      259            Massachusetts      6620       89                  Florida      6338       77                 Illinois      5994       99               Washington      5305      222                Louisiana      5237      239             Pennsylvania      4963       63                  Georgia      3815      111                    Texas      3726       53              Connecticut      3128       69                 Colorado      2627       51                Tennessee      2391       23                  *Others     25475      462 Creating datafile '2020-03-31_2221Z.csv'...       315 rows written (  855007 /  42032 /  177857 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ The format difference is due to the fact that Admin2 reporting is by county, but doesn't populate Recovered and Active, while the Admin1 reporting includes Recovered (which would allow Active to be computed, but Recovered is now a lump-sum record in the ncov_cases service, so in practice, Active and Recovered are not available). So, how do I know this script populates the same data shared on GitHub?  Well, I wrote a validator utility to create two CovidSummary objects, then iterate from one searching the other for duplicate keys, and reporting missing rows, data mismatches, and unmatched rows, and the output was: D:\covid-19>python validate.py‍‍ ======================================================================== 0 errors / 3429 lines‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Okay, I cheated a bit, since the daily JHU CSV reports that the Diamond Princess and Grand Princess cruise ships are docked on Null Island, I treated 0.0 degrees latitude/longitude as a wildcard that matches any coordinate, and I only compare coordinates to 5 places, since string comparison doesn't work well with floating-point values, but everything else has aligned perfectly.  UPDATE @ 2100 EST: Unfortunately, sometime today, the US Admin2 jurisdictions without any confirmed cases were deleted from the ncov_cases_US service feed, the FIPS code displayed for the Northern Mariana Islands and US Virgin Islands disappeared, and the US territory without any confirmed cases (American Samoa) also disappeared (along with its FIPS code).  I've tweaked the exporter to conform to this, and it runs cleanly now: D:\covid-19>python validate.py‍‍ ======================================== sum1 = data2\2020-03-31.csv sum2 = data2\2020-04-01_0110Z.csv 0 errors / 2434 lines‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ But I'm expecting Guam to have its FIPS code removed in the coming days. Since the validation tool implements Python classes to parse and search both old-style and new-style CSV files, I've attached that as well (as validator-v10.zip) My next task is to write some code to exploit this near real-time data resource, and use it to maintain PostgreSQL tables to produce clones of the ncov_cases* services' data. Now attached is jhuEmulator-v20.zip -- Changelog: Implemented adminLevel=2 with multiple queries to replicate three-level results in GitHub CSV files Added --confirmedOnly flag to skip counties without Confirmed cases (obviated by feed change) Added --topStates to report ordered state-wide impact (requires --verbose=True and non-zero --adminLevel) Added FIPS lookup for US Possessions (Note: Puerto Rico = None) Ignored US admin1 summary values in admin2 feed (2-digit FIPS) Added validateKey to correct Combined_Key with missing spaces after commas Removed FIPS from US possessions and suppressed US possessions without confirmed cases (as per 2020-03-31.csv) == Update 01-Apr @ 0100 EST == Long-running service execution is great for finding bugs... Now attached is jhuEmulator-v21.zip -- Changelog: Fixed significantChange field comparison error with Admin1 field names in Admin2 service == Update 01-Apr @ 0920 EST == Overnight service execution is great for finding bugs, but errors that arise after several hours are less fun... Now attached is jhuEmulator-v22.zip -- Changelog: Fixed significantChange field comparison error for *all* Admin1 field names in the Admin2 service Fixed paren alignment defect in error handling in pullData Also attached is covidValidate_v11.zip -- Changelog: Tightened CovidReport.compare() loop Fixed range check defect in main == Update 01-Apr @ 2030 EST == The server glitched, dropping all US data, and I got to exercise some error code that doesn't normally see traffic, which I've now tweaked to be more resilient. Attached is jhuEmulator-v23.zip -- Changelog: Fixed TypeError when service result is empty (None) == Update 08-Apr @ 1600 EST == I tweaked the exporter to always write data at UTC midnight, so that the validator has the same Last_Update value for comparison with the JHU published file. I had to improve date parsing in the CovidReport class (the corrected 2020-04-06.csv has a different date format). I'm still seeing some JHU data with incorrect 000xx FIPS codes, and the Admin1 and Admin2 feeds are out of sync for the Northern Mariana Islands, but otherwise looking good. Attached is jhuEmulator-v24.zip -- Changelog: Tweak to force 0000Z download, even when --skipTrivial is enabled (for comparison with GitHub published files) Also attached is covidValidate_v12.zip -- Changelog: Cleaned up CovidReport.extract() to handle bad formatting gracefully and to support the 'm/d/y HH:MM' date used in the revised 2020-04-06.csv datafile Loosened the Last_Update comparison code so that second- truncated timestamps can compare successfully   == Update 12-Apr @ 2320 EST == The JHU daily CSV format changed again, adding five more fields (and moving FIPS later in the display order). I've got the covidValidate.py handling the new format, but am not yet generating it... Attached is covidValidate_v30.zip -- Changelog: Count, but don't display, Combined_Key mismatches Added parsing (but not yet validation) of fields added to 04-12-2020.csv - V ... View more

Like many, I've been using the daily data from the Johns Hopkins University Ops Dashboard site, but I've been frustrated by the periodicity and timing -- Midnight UTC is all well and good, but then you've got 12+ hour old data before folks start their workday on the US East Coast. But there is another route to the data, since it's published as a feature service. So Sunday night I whipped up a Python script to pull the full feature feed from AGOL, and write an hourly file in the same format.  This script is plain-vanilla Python (no arcpy), and has been tested on both Python 2.7.16 and 3.8.2. Execution looks looks something like this: D:\covid-19>python jhuEmulator.py DELAY --verbose=True --interval=2h Sleeping 35.67 minutes... Querying feature service... 155.6 KB retrieved (310ms elapsed) Creating datafile '2020-03-25_0600Z.csv'... 300 rows written ( 422989 / 18916 / 108578 ) Sleeping 2.00 hours... Querying feature service... 155.6 KB retrieved (179ms elapsed) Creating datafile '2020-03-25_0800Z.csv'... 300 rows written ( 423670 / 18923 / 108629 ) Sleeping 2.00 hours... Querying feature service... 155.6 KB retrieved (180ms elapsed) Creating datafile '2020-03-25_1000Z.csv'... 300 rows written ( 425493 / 18963 / 109191 ) Sleeping 2.00 hours... Querying feature service... 156.1 KB retrieved (146ms elapsed) Creating datafile '2020-03-25_1200Z.csv'... 301 rows written ( 435006 / 19625 / 111822 ) Sleeping 2.00 hours... Exiting on 'stop.now' request D:\covid-19>dir/b data 2020-03-25_0600Z.csv 2020-03-25_0800Z.csv 2020-03-25_1000Z.csv 2020-03-25_1200Z.csv D:\covid-19>head data\2020-03-25_1200Z.csv Province/State,Country/Region,Last Update,Confirmed,Deaths,Recovered,Latitude,Longitude ,Italy,2020-03-25T07:24:40,69176,6820,8326,41.87194,12.56738 Hubei,China,2020-03-25T02:43:02,67801,3163,60810,30.9756,112.2707 ,Spain,2020-03-25T07:24:40,47610,3434,5367,40.463667,-3.74922 ,Germany,2020-03-25T07:24:40,34009,172,3532,51.165691,10.451526 ,Iran,2020-03-25T07:24:40,27017,2077,9625,32.427908,53.688046 New York,US,2020-03-25T07:24:54,26376,271,0,42.165726,-74.948051 ,France,2020-03-25T07:24:40,22304,1100,3281,46.2276,2.2137 ,Switzerland,2020-03-25T07:24:40,10171,135,131,46.8182,8.2275 ,Korea, South,2020-03-25T07:24:40,9137,126,3730,35.907757,127.766922‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Then I worried about system load (One ought not pound the free service that one wants to use into oblivion) and update frequency (Wouldn't it be nice to get updates as they happen?), so I started working on tweaks: Added argparse command-line switches to change execution parameters (vice hacking the source) Added 0-20 random seconds delay for execution of the query (so synchronized servers don't overload the feature service, yet with plenty of time to finish within the targeted minute) Permitted a directory tree archiving style (creating directories if necessary) I started working on a way to only write a dataset which has had a substantive change over the previous timestamp (so the timing could be shortened to, say, 5 minutes, without needlessly transmitting files), but I discovered I can't use the Last_Update field for this (since the timestamp changes without case count changes), and then the CSV data format posted on GitHub changed Monday night to include Admin2 (county-level) data in the US (at lower resolution -- no Recovered/Active values), so I'll post this as-is for reference (or for those that now need an alternate source for data in the old format) and figure out how to model the new format... Note that there are three modes (you need to choose one): ONCE - Create one file, now, and exit (suitable for a cron job or other batch-oriented invocation at an interval greater than one hour) IMMEDIATE - Create one file now, and then continue file creation based on the interval modulus (default hourly) DELAY - Create first file in next expected time window (if the old job was killed, and you just want to pick up where left off) It's possible to place files in a directory tree based on the time format.  For example, if you specify    --csvFormat="y%Y\m%m\d%d\t%H%MZ.csv" Then the yYYYY\mMM\dDD folder will be created if necessary, and the "tHHMMZ.csv" data file added to that. You can gracefully exit the application by creating a "stop.now" file in the active directory (or creating the file referenced by the optional --stopPath flag). == Update 25-Mar @ 0800 EST == Now attached is jhuEmulator-v11.zip -- Changelog: Parameterized URL assembly for future US feed support Added mapping for missing fields (to None value) Fixed stopPath utilization defect Fixed sys.exit() defect Reduced wait for stopPath detection Expanded interval to even divisions of hour and day (2m,3m,4m,5m,6m,10m,12m,15m,30m,1h,2h,3h,4h,6h,12h) I should probably re-emphasize that this tool can be used to preserve the "old format" feed until your tools have the ability to process the new format with FIPS and Admin2 fields. == Update 25-Mar @ 0840 EST == Now attached is jhuEmulator-v12.zip -- Changelog: Fixed IMMEDIATE computeDelay() defect Note that the stop file will now (as of v11)  gracefully exit the app within a minute of creation. == Update 25-Mar @ 1520 EST == Now attached is jhuEmulator-v13.zip -- Changelog: Detect low-volume data pulls and skip write operation Fixed defect in byVolume sorting (for/else) Fixed defect in pull error logic 0.4 millisecond/existence test timing compensation Made Interrupt handling in sleep cleaner Working on nominal change detection and US counties feed support next... == Update 25-Mar @ 1820 EST == Now attached is jhuEmulator-v14.zip -- Changelog: Added substantive change detection (only write if case counts or Lat/Lon are different, not if just Last_Update changed) - Uses new --skipTrivial flag (default False) Fixed naming defect in main loop Self-initializing record count expectation The --skipTrivial flag is what I had been looking to achieve in near-realtime feed support. With it, you can set a short interval (though not less than 2 minutes) and have CSV files only written when "substantive change" is detected (row count or changes to Confirmed, Deaths, or Reported field values, mostly).  Executing it with --interval=2m (which is probably way too often -- 5, 6, 10, 12, or 15 would be better) results in output like this: D:\covid-19>python jhuEmulator.py IMMEDIATE ^ More? --verbose=True --skipTrivial=True --interval=2m Querying feature service... 156.6 KB retrieved (232ms elapsed) Creating datafile '2020-03-25_2207Z.csv'... 302 rows written ( 464026 / 20946 / 113691 ) Sleeping 0.15 minutes... Querying feature service... 156.6 KB retrieved (152ms elapsed) Skipping write of trivial change to '2020-03-25_2208Z.csv'... Sleeping 2.07 minutes... Querying feature service... 156.6 KB retrieved (195ms elapsed) Skipping write of trivial change to '2020-03-25_2210Z.csv'... Sleeping 2.16 minutes... Querying feature service... 156.6 KB retrieved (176ms elapsed) Skipping write of trivial change to '2020-03-25_2212Z.csv'... Sleeping 1.80 minutes... Querying feature service... 156.6 KB retrieved (187ms elapsed) Skipping write of trivial change to '2020-03-25_2214Z.csv'... Sleeping 2.20 minutes... Querying feature service... 157.2 KB retrieved (185ms elapsed) Creating datafile '2020-03-25_2216Z.csv'... 303 rows written ( 466836 / 21152 / 113769 ) Sleeping 1.88 minutes... Querying feature service... 157.2 KB retrieved (179ms elapsed) Skipping write of trivial change to '2020-03-25_2218Z.csv'...‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ My next goal is to write data in the new (post 23-Mar) JHU CSV format from the ..._US service.  I'll likely set it up to be both global and US-only. Please let me know if you use this or encounter problems... == Update 25-Mar @ 2200 EST == Blog title updated. == Update 26-Mar @ 1800 EST == Now attached is jhuEmulator-v15.zip -- Changelog: Added time prefix to non-verbose output Added --adminLevel=[0,1] flag to permit country summaries Added --usOnly support to restrict results to US data Fixed ONCE defect when retrieval fails Fixed CSV quoting bug with Python 3 Note: adminLevel=2 is partially functioning, but disabled until a work-around for the 1000 feature limit is added Good news: jhuEmulator has code to export new-style format files Bad news:   Only 1000 rows can be received per query I did add the --adminLevel flag, but right now it only functions to summarize the countries that are usually distributed with province/state level data (US, Canada, China, and Australia). Python 3 users: Note that v1.5 fixes a formatting defect that failed to place double-quotes around values with embedded quotes. The next snapshot should be v2.0, which will assemble multiple queries into a single file which approximates the new-format CSV style. == Update 29-Mar @ 2330 EST == Now attached is jhuEmulator-v16.zip -- Changelog: Fixed timezone defect in datetime.datetime.fromtimestamp call (Last_Update had been captured in local timezone, not UTC) Added date display in non-verbose operation at midnight UTC I'm still working on the Admin2 format support. It's nearly done, but not ready for release. I discovered an ugly bug in the v15 and earlier extraaction -- the UTC dates in Last_Update are recorded in the local timezone, not UTC, so I wanted to get this fix out there, BUT I'm now showing far more cases than the JHU Ops Dashboard, so the code that was supposed to accumulate admin2 data into an admin1 summary seems to be double-counting.  The same issue exists in v15 and earlier (and v20, right now), so I'll need to figure out what went wrong... == Update 29-Mar @ 2330 EST == Heh. My Chrome session needed refresh. The total in the website correlates to the extraction feed. For what it's worth, here's the log of my last 26 hours of data extraction, pulling data every 15 minutes, but only saving if if there was non-trivial change: D:\covid-19>python jhuEmulator.py DELAY --interval=15m --skipTrivial=True 0245: 312 rows written ( 663828 / 30822 / 139451 ) 0300: 312 rows written ( 664608 / 30846 / 140156 ) 0400: 312 rows written ( 664695 / 30847 / 140156 ) 0515: 312 rows written ( 664924 / 30848 / 140222 ) 0630: 312 rows written ( 665164 / 30852 / 140225 ) 0745: 312 rows written ( 665616 / 30857 / 141746 ) 0845: 312 rows written ( 666211 / 30864 / 141789 ) 0945: 312 rows written ( 669312 / 30982 / 142100 ) 1100: 312 rows written ( 678720 / 31700 / 145609 ) 1200: 312 rows written ( 679977 / 31734 / 145625 ) 1315: 312 rows written ( 681706 / 31882 / 145696 ) 1430: 312 rows written ( 684652 / 32113 / 145696 ) 1530: 312 rows written ( 685623 / 32137 / 145706 ) 1645: 312 rows written ( 691867 / 32988 / 146613 ) 1815: 312 rows written ( 704095 / 33509 / 148824 ) 1900: 312 rows written ( 710918 / 33551 / 148900 ) 2000: 312 rows written ( 713171 / 33597 / 148995 ) 2130: 312 rows written ( 716101 / 33854 / 149071 ) 2230: 312 rows written ( 718685 / 33881 / 149076 ) 2345: 312 rows written ( 720117 / 33925 / 149082 ) ----> Mon 2020-03-30 UTC 0045: 312 rows written ( 721584 / 33958 / 149122 ) 0200: 312 rows written ( 721817 / 33968 / 151204 ) 0300: 312 rows written ( 722289 / 33984 / 151901 ) 0400: 312 rows written ( 722435 / 33997 / 151991 )‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ == Update 31-Mar @ 2120 EST == The Admin2 format emulator jhuEmulator-v20.zip is now published in the new blog post:       https://community.esri.com/people/vangelo-esristaff/blog/2020/04/01/python-utility-to-write-covid-19-csv-snapshots-from-arcgiscom-feature-services-v2  There were output changes only today, so v2.0 is likely to to have some updates to keep in sync. The most significan change for anyone using the v1.x tools is that the new default file format is the post-03/23 format, but that can be overridden with --adminVersion=1. - V ... View more