Latest Contributions by ManushiMajumdar

This week we look at ways to extract data from social media platforms like Twitter. Over the years, Twitter has proved to be a gold mine for learning about events happening in real time and the social reach and impact of those events. Most people tweet about their opinion or experience regarding an event by tagging and mentioning relevant twitter handles (accounts) or by using hashtags that categorize their tweet and helps users find content related to the hashtag or topic. This data can be used for research as well as to obtain civic and demographic insights. Accessing publicly available tweets for our city or local region helps invert our lens and understand the general public and what matters to them. We will show you 5 different techniques of extracting and filtering data from Twitter, and to publish it ArcGIS making it usable within several compelling apps. Curious? Click here to learn more. ... View more

This week we shift gears to analyze a data catalog for a Hub. A Data Catalog, simply put, is an organized inventory of data assets and their metadata across all the data sources in your Hub. Click here to read more about this in our blog post. We have set up two notebooks to analyze data catalogs with different sources of data. One notebook can be used to analyze basic summary of any data on a Hub, even those with data distributed across multiple different platforms (including the ArcGIS platform). We demonstrate this notebook with the City of Seattle's data catalog. The other notebook can be used to analyze both basic and advanced summary of ArcGIS-hosted data on a Hub, i.e. one with all of its data on the ArcGIS platform. We demonstrate this second notebook with the data catalog of Washington, DC. Links to Notebooks: Summarizing data assets of a Hub Summarizing data assets of a Hub (advanced) ... View more

This week we explore how charting libraries in Python can help us visualize temporal and spatial patterns to get a quick snapshot of our city. For this, we look at the crime data in Washington, DC as well as 311 service requests from the year 2019 in two separate notebooks. We start by extracting the month, hour and day of the week for each record from the timestamp field. We plot these on histograms to observe a quick visual of which months/hours/days have most number of recorded criminal incidents and 311 calls. This helps us identify the time periods of high and low activity. We create small multiple charts for criminal incidents based on the SHIFT of the day they were reported in, i.e. day, evening or midnight. We then proceed to create these charts for offense type, to get a sense of density and spread of these incidents based on offense type. For 311 calls, we plot small multiple horizontal bar charts to summarize the most frequent category of requests for each Ward (administrative boundary) of Washington, DC along with a chart of small multiple maps for the agency most calls are routed to. Using these small multiple charts and maps we can compare how spatial patterns differ within the city, to understand sub-spatial nuances better. https://arcgishub.maps.arcgis.com/home/item.html?id=0e86902149314b98b6715831dd1bf35a#preview We conclude with a statistical analysis that evaluates if number of crimes (or 311 calls) reported in each census tract of a neighborhood, correlate with the population of those census tracts. The steps followed are: Enrich the most recent census tracts layer for Washington, DC with recent population data. Join the neighborhoods layer to this data spatially (based on location of the two), in order to get the appropriate neighborhood(s) that each census tract falls within. Compute a Pearson correlation coefficient for the number of crimes (or 311 calls) reported and population for each census tract of a neighborhood. Links to notebooks – Temporal, Spatial and Statistical patterns in civic (crime) data Temporal, Spatial and Statistical patterns in civic (311 service requests) data Accompanying blogpost ... View more

This week w e will look at a simple Binary classification technique which normally classifies your data to yes/no, true/false, this/that categories. We evaluate if zoning plays a role in building permits being revoked . To do this we start by reading in the Building Permits data for Miami, FL since 2014 and filter the data to include only those permits that are still approved and those that have been revoked.   Normally , issued building permits get revoked/cancelled later if the scope of work changes from what was initially estimated and applied for. We see if that is a factor, and if yes how strong a factor in successfully classifying our permits as revoked or not. We split our data into a 75% training set which is used to build our classifier. The remaining 25% of data is used to test how well our classifier performs. Depending on the application at hand, training and test data may not be derived from the same source but in the interest of these notebooks we will work with the same data. We use the sklearn library of Python to build this classifier and to evaluate its performance. Don't forget to check out our notebook to see what quantified impact zoning has on permits being revoked.   A binary classifier can be built to answer several questions with your civic data. To name a couple, you can evaluate   if a building/restaurant will pass inspection,   if a 311 citizen service request will be closed within a certain timeframe or not   Link to notebook - Does zoning play a role in building permits being revoked?   Accompanying blogpost ... View more

In this notebook we work with the Vision Zero street safety survey from Washington, DC. This data is collected through a web application where the public can select a particular street segment and convey their concerns about its safety. We read this data in using the ArcGIS API for Python. Following that we use four Python libraries that assist us with the text processing and analysis - WordCloud , nltk , textblob and spacy . Using WordCloud , we first create a word cloud of the most popular words in the survey. We then import nltk to identify the words of high frequency and high relevance to the survey and regenerate a word cloud. This gives us a quick visual snapshot of what people are speaking about the most. Having done that we extract the most popular words mentioned, that suggest the topics of importance in the survey. We proceed to calculate the sentiment score for each comment, ranging from -1 (negative sentiment) to +1 (positive sentiment) using textblob and visualize the distribution of the scores. This gives us a general sense of what the citizens feel about the safety of the streets. We also extract the top 10 positive and negative comments, based on their sentiment scores to get a sense of the comments with strong opinions. We then conclude with a final technique that uses spacy to identify the named entities or proper nouns mentioned in these comments and classify them to identify if they are names for a person, place, organization, etc. This is a useful technique to quickly extract the subject and focus of a comment. Link to notebook - Exploratory text analysis of comments from surveys Accompanying blog post ... View more

This week we focus on the attributes of a dataset to understand how relationships between attributes can be detected and interpreted. We also extend that understanding further to spot hidden patterns in our data. In the first example we fetch neighborhood boundaries for Washington, DC to observe correlation in socioeconomic factors. We enrich the neighborhoods layer with a few socioeconomic variables such as, variables for Population, Median Household Income, Households below poverty levels, to name a few. We then display the data as a scatter matrix - a collection of scatter plots - which compare the relation of each numerical variable with the other to see if changes in one variable reflect as changes in the other variable in some way. Having obtained a visual understanding of these correlated variable pairs, we then use statistical tests from the scipy (Scientific Python) library of Python to numerically compute this correlation for a few variable pairs. The second notebook demonstrates two different techniques of detecting clusters or patterns in data. We begin by fetching data for rodent inspection and treatment sites in Washington, DC for the last 30 days to detect point clusters if any, which helps inform strategies for follow-up treatments and inspections. The second example we look at checks to see if neighborhoods within the city of Tucson can be grouped together based on similarities in income variables. We read in data and then extract variables of interest in a separate dataframe. This data is used as the input for the KMeans unsupervised learning method from the scikit-learn library of Python. This helps us detect neighborhood clusters that exhibit similarity in our variables of choice. ... View more