Take the column names from the NYC_property_sales data frame, and then update all column names to replace all spaces with underscores, and then update all column names to lower case. Note we saved this dataset with the variable name brooklyn for future use. 4. View the Data with tidyr::glimpse() Crystal Lewis gave a presentation to R-Ladies St. Louis recently on the topic of cleaning data in R. Her slides and materials are available on GitHub. Those of us who work with data are professionals. Working with data is one of the main skills for which we are hired. These are not skills that come naturally, and so it should not be surprising that those without our training and experience provide data we consider to be "messy." When people use highlighting in spreadsheets, for example, they are not doing anything wrong. They are working with their data in a way that makes most sense to them. That this method of working with data doesn't lend itself to the types of analysis we do is a secondary consideration (if it is a consideration at all). The type of tidy data that many of us like to work with works for our purposes, but it would likely be hard for others to make sense of. Different horses for different courses.

Karl Broman and Kara Woo's 2018 article titled Data Organization in Spreadsheets has tons of great tips. The abstract lays out several of them: Messy datasets are everywhere. If you want to analyze data, it’s inevitable that you will need to clean data. In this tutorial, we're going to take a look at how to do that using R and some nifty tidyverse tools. Note that you could also replace median in the formula with mean to instead replace missing values with the mean value of each column. If we combined these dataframes and ended up with more columns than we had in the brooklyn dataframe, it could indicate a problem such as an erroneous column name in one of the datasets. But that did not happen here, so we can move on to cleaning up column names. 9. Clean Up Column Names with magrittr Magic! Notice the dramatic drop in property sales in April, 2020. Might this related to the COVID-19 pandemic? As you can see, with only a few lines of code, we can begin to explore our data and ask some interesting questions!The glimpse() function provides a user-friendly way to view the column names and data types for all columns, or variables, in the data frame. With this function, we are also able to view the first few observations in the data frame. This data frame has 20,185 observations, or property sales records. And there are 21 variables, or columns. 5. Data Types

It’s useful that SALE DATE is stored in a format that represents calendar dates and times because this enables us to use a single line of code to make a histogram of property sales by date: qplot( SALE DATE, data = brooklyn) remove_empty(): “Removes all rows and/or columns from a data.frame or matrix that are composed entirely of NA values.” Now that tidyverse is loaded into memory, take a “glimpse” of the Brooklyn dataset: glimpse(brooklyn) ## Observations: 20,185 GROSS.SQUARE.FEET and SALE.PRICE are also stored as factors. We can’t perform arithmetic operations, like calculating the mean, on a factor!The tidyverse tools provide powerful methods to diagnose and clean messy datasets in R. While there's far more we can do with the tidyverse, in this tutorial we'll focus on learning how to: Notice that the new data frame does not contain any rows with missing values. Example 2: Replace Missing Values with Another Value When people use highlighting in spreadsheets, for example, they are not doing anything wrong. They are working with their data in a way that makes most sense to them. It is the same with data science projects. If your data is poorly prepped, unreliable results can plague your work no matter how cutting-edge your statistical artistry may be. Which, for anyone who translates data into company or academic value for a living, is a terrifying prospect. Notice that all of the data frames have been cleared from the environment but all of the other objects remain. Additional Resources