Latest Contributions by ManushiMajumdar

This week we explore how charting libraries in Python can help us visualize temporal and spatial patterns to get a quick snapshot of our city. For this, we look at the crime data in Washington, DC as well as 311 service requests from the year 2019 in two separate notebooks. We start by extracting the month, hour and day of the week for each record from the timestamp field. We plot these on histograms to observe a quick visual of which months/hours/days have most number of recorded criminal incidents and 311 calls. This helps us identify the time periods of high and low activity. We create small multiple charts for criminal incidents based on the SHIFT of the day they were reported in, i.e. day, evening or midnight. We then proceed to create these charts for offense type, to get a sense of density and spread of these incidents based on offense type. For 311 calls, we plot small multiple horizontal bar charts to summarize the most frequent category of requests for each Ward (administrative boundary) of Washington, DC along with a chart of small multiple maps for the agency most calls are routed to. Using these small multiple charts and maps we can compare how spatial patterns differ within the city, to understand sub-spatial nuances better. https://arcgishub.maps.arcgis.com/home/item.html?id=0e86902149314b98b6715831dd1bf35a#preview We conclude with a statistical analysis that evaluates if number of crimes (or 311 calls) reported in each census tract of a neighborhood, correlate with the population of those census tracts. The steps followed are: Enrich the most recent census tracts layer for Washington, DC with recent population data. Join the neighborhoods layer to this data spatially (based on location of the two), in order to get the appropriate neighborhood(s) that each census tract falls within. Compute a Pearson correlation coefficient for the number of crimes (or 311 calls) reported and population for each census tract of a neighborhood. Links to notebooks – Temporal, Spatial and Statistical patterns in civic (crime) data Temporal, Spatial and Statistical patterns in civic (311 service requests) data Accompanying blogpost ... View more

This week w e will look at a simple Binary classification technique which normally classifies your data to yes/no, true/false, this/that categories. We evaluate if zoning plays a role in building permits being revoked . To do this we start by reading in the Building Permits data for Miami, FL since 2014 and filter the data to include only those permits that are still approved and those that have been revoked.   Normally , issued building permits get revoked/cancelled later if the scope of work changes from what was initially estimated and applied for. We see if that is a factor, and if yes how strong a factor in successfully classifying our permits as revoked or not. We split our data into a 75% training set which is used to build our classifier. The remaining 25% of data is used to test how well our classifier performs. Depending on the application at hand, training and test data may not be derived from the same source but in the interest of these notebooks we will work with the same data. We use the sklearn library of Python to build this classifier and to evaluate its performance. Don't forget to check out our notebook to see what quantified impact zoning has on permits being revoked.   A binary classifier can be built to answer several questions with your civic data. To name a couple, you can evaluate   if a building/restaurant will pass inspection,   if a 311 citizen service request will be closed within a certain timeframe or not   Link to notebook - Does zoning play a role in building permits being revoked?   Accompanying blogpost ... View more

In this notebook we work with the Vision Zero street safety survey from Washington, DC. This data is collected through a web application where the public can select a particular street segment and convey their concerns about its safety. We read this data in using the ArcGIS API for Python. Following that we use four Python libraries that assist us with the text processing and analysis - WordCloud , nltk , textblob and spacy . Using WordCloud , we first create a word cloud of the most popular words in the survey. We then import nltk to identify the words of high frequency and high relevance to the survey and regenerate a word cloud. This gives us a quick visual snapshot of what people are speaking about the most. Having done that we extract the most popular words mentioned, that suggest the topics of importance in the survey. We proceed to calculate the sentiment score for each comment, ranging from -1 (negative sentiment) to +1 (positive sentiment) using textblob and visualize the distribution of the scores. This gives us a general sense of what the citizens feel about the safety of the streets. We also extract the top 10 positive and negative comments, based on their sentiment scores to get a sense of the comments with strong opinions. We then conclude with a final technique that uses spacy to identify the named entities or proper nouns mentioned in these comments and classify them to identify if they are names for a person, place, organization, etc. This is a useful technique to quickly extract the subject and focus of a comment. Link to notebook - Exploratory text analysis of comments from surveys Accompanying blog post ... View more

This week we focus on the attributes of a dataset to understand how relationships between attributes can be detected and interpreted. We also extend that understanding further to spot hidden patterns in our data. In the first example we fetch neighborhood boundaries for Washington, DC to observe correlation in socioeconomic factors. We enrich the neighborhoods layer with a few socioeconomic variables such as, variables for Population, Median Household Income, Households below poverty levels, to name a few. We then display the data as a scatter matrix - a collection of scatter plots - which compare the relation of each numerical variable with the other to see if changes in one variable reflect as changes in the other variable in some way. Having obtained a visual understanding of these correlated variable pairs, we then use statistical tests from the scipy (Scientific Python) library of Python to numerically compute this correlation for a few variable pairs. The second notebook demonstrates two different techniques of detecting clusters or patterns in data. We begin by fetching data for rodent inspection and treatment sites in Washington, DC for the last 30 days to detect point clusters if any, which helps inform strategies for follow-up treatments and inspections. The second example we look at checks to see if neighborhoods within the city of Tucson can be grouped together based on similarities in income variables. We read in data and then extract variables of interest in a separate dataframe. This data is used as the input for the KMeans unsupervised learning method from the scikit-learn library of Python. This helps us detect neighborhood clusters that exhibit similarity in our variables of choice. ... View more

This week we look at two spatial analysis techniques in ArcGIS. We use data from Connecticut to investigate Which housing districts need more Covid-19 testing sites?. In this analysis, we fetch housing districts of the state and add variables that provide us with total population and population density (as of 2019) for each housing district. Having pulled in data for the existing Covid 19 testing site locations, we then compute areas that are drivable within 15 minutes to these clinics, to identify housing districts that are under-served by quick access to testing locations relative to their population. This analysis could be used to answer several other questions, for instance, 'Are certain school districts under-served by quick access to public libraries?' or 'Which neighborhoods need more parks relative to the number of children in the neighborhood?' In the second notebook, we will work with census tracts of Atlanta to answer the question Which are the census tracts with unusually high unemployment rates?. For the population age 16+, we use the percentage of unemployed people in each census tract to identify statistically significant outliers to identify census tracts with unusually high unemployment rates compared to the neighboring census tracts. This technique can help address questions such as 'Do we observe areas of unexpectedly low mortality rate or unexpected disease outbreak?' or 'Where can we notice distinct difference in high or low crime zones relative to the neighboring areas?', to name a few. I encourage you to download these notebooks and use them to find data-driven answers to your questions about your region. I can't wait to connect with you on your findings or questions. Feel free to comment below with questions, or analyses with civic data that you would like for us to explore and demonstrate. ... View more

With ArcGIS Hub's Civic Analytics notebook series we will empower you with spatial and statistical analysis techniques to work with civic data using newly released ArcGIS Notebooks. The goal is to put together a collection of notebooks where we will address a particular question or topic using data science tools in Python, and to take it forward by inspiring a community of citizens to use these notebooks as a means to understand their local data and region better and share the results of their data analysis experiments to further this pursuit of building a data-informed community. This week we start with notebooks that focus on exploring and understanding the spatial and temporal aspects of street crashes in Ottawa, Canada in 2018. We focus on fetching data and highlighting the distribution of crashes across space and time to gain a more visual perception of civic data beyond the usual tabular structure. We also have a guide to walk you through the process of finding and using data. Additionally, for those looking here is a great interactive Python tutorial and resource for the ArcGIS API for Python. Click here to read more on our blog post. Feel free to share your thoughts and results from your experiments with civic data from your local Hub in these notebooks! ... View more

Hub's Python API is a Python library (arcgishub) that allows users to interact with their Hub and its items. The goal of the API is to empower Hub's users with Python methods to automate their Hub workflows and easily perform analysis in context of their Hub's content. Currently the library allows users to create and manipulate initiatives, sites, events and indicators. For sites, a user can create, fetch, search, update (item) and delete sites in Python. This API also supports cloning sites and initiatives across Hub Premium, Basic organizations as well as ArcGIS Enterprise deployments, as applicable. We are looking to provide support for Site Editing and would love to hear your thoughts and needs. Programmatically editing aspects of a site can allow mass editing of sites in an organization (e.g. edit a particular text card to reflect new Mayor name across all sites). Other potential uses of site editing in Python may involve adding/deleting new rows and cards to your site layout. Similarly, we would like to learn more about your requirements and your specific workflows for site editing that you would prefer automating with the Python API. Understanding your requirements will help us continue building a lovable Hub for developers. Go ahead, and make your 'site editing in Python' wishes! ... View more