```{r}
-#| label: setup
-#| message: false
-
-library(tidyverse)
-library(janitor)
-``````{r}
+#| label: setup
+#| message: false
+
+library(tidyverse)
+library(janitor)
+```The data outputs in this book might differ from what you get since the source data is updated every week. This is especially true in videos and gifs.
+The data outputs in this book might differ from what you get since the source data is updated every week. This is especially true in videos, gifs and takeaways.
+This chapter won’t have data past 2024.
@@ -496,8 +503,8 @@Since we are starting a new notebook, we need to set up a few things. First up we want to list our goals.
@@ -596,7 +603,7 @@Rows: 344,300
+Rows: 346,600
Columns: 7
$ chart_date <date> 1958-08-04, 1958-08-04, 1958-08-04, 1958-08-04, 1958-08…
$ current_rank <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 1…
@@ -704,12 +711,12 @@
Basically we are summarizing the total number of rows in the data. When this book was last updated there were 344300 rows.
+Basically we are summarizing the total number of rows in the data. When this book was last updated there were 346500 rows.
But I bet you’re thinking: “Yo Professor, you said we want to count the number of times a performer has appeared, right?”
This is where we bring in a close friend to summarize() … the group_by() function.
performer names, and as such considers collaborations separately. For instance, Drake is near the top of the list but those are only songs he performed alone and not the many, many collaborations he has done with other performers. Songs by “Drake” are counted separately than “Drake featuring Future” and even “Future featuring Drake”. You’ll need to make this clear when you write about this later. (If you include collaborations, Drake has way more appearances than Taylor.)head() to “cut” this list is not the best method because there could be a tie between the 10th and 11th record. In the future we’ll learn filter() to do this better. But head() is pretty useful when you just need to see the top of your data.performer names, and as such considers collaborations separately. For instance, Drake is second on the list but those are only songs he performed alone and not the many, many collaborations he has done with other performers. Songs by “Drake” are counted separately than “Drake featuring Future” and even “Future featuring Drake”. You’ll need to make this clear when you write about this later. (If you include collaborations, Drake has way more appearances than Taylor.)head() to “cut” this list is not the best method because there could be a tie between the 10th and 11th record. In the future we’ll learn filter() to do this better. But head() is pretty useful when you just need to see the top of your data.Data Takeaway: Taylor Swift has more solo appearances on the Billboard Hot 100 than any other artist in history with 1,582 appearances through July 27, 2024. This does not include collaborations with other artists.Data Takeaway: Taylor Swift has more solo appearances on the Billboard Hot 100 than any other artist in history with 1,600 appearances through December 28, 2024. This does not include collaborations with other artists.Writing these sentences help you understand what you’ve found in your data. When you get toward the end of your project, you’ll collect these takeaways in your index.qmd file.
The value we use to cut the list of – 65 in this case – is arbitrary. We just want to choose a value that makes common sense. 70 seemed too high because that would yield only five or so songs, and 60 was too low because there are lots of ties in the lower 60s.
The value we use to cut the list of – 65 in this case – is arbitrary. We just want to choose a value that makes common sense. 70 seemed too high because that would yield 7 songs, and 60 was too low because there are lots of ties in the lower 60s.
Data Takeaway: The song "Heat Waves" by Glass Animals has appeared 91 times on the Billboard Hot 100 chart, more than any other song through July 27, 2024. "Blinding Lights" by The Weeknd is second with 90 appearances.Data Takeaway: The song "Heat Waves" by Glass Animals has appeared 91 times on the Billboard Hot 100 chart, more than any other song through December 28, 2024. "Blinding Lights" by The Weeknd is second with 90 appearances.I’m providing these takeaways in this chapter as examples. You’ll be asked to write your own in future projects.
Now you have the answers to the song with the most weeks at No. 1 with a logical cutoff. If you add to the data later, that logic will still hold and not cut off arbitrarily at a certain number of records. For instance, Mariah Carey’s All I Want for Christmas will likely reach its 15th week at the top of the charts in the 2024-25 holiday season.
+Now you have the answers to the song (or songs) with the most weeks at No. 1 with a logical cutoff. If you add to the data later, that logic will still hold and not cut off arbitrarily at a certain number of records. For instance, Mariah Carey’s All I Want for Christmas could eventually top this list because that song reaches No. 1 for a week or two every holiday season.
Data Takeaway: The song "Old Town Road" by Lil Nas X Featuring Billy Ray Cyrus has spent more time at the top of the Billboard Hot 100 than any other song in history at 19 weeks.Data Takeaway: Both Lil Nas X's "Old Town Road" and Shaboozey's "A Bar Song (Tipsy)" have spent the most time at the top of the Billboard Hot 100 through December 28, 2024 with 19 appearances.hot100 |>
- filter(chart_date >= "2019-01-01") |>
+ filter(chart_date >= "2020-01-01") |>
head() # added just to shorten our resulthot100 |>
filter(
current_rank == 1,
-1 chart_date >= "2019-01-01"
+1 chart_date >= "2020-01-01"
) |>
distinct(title, performer) |>
group_by(performer) |>
@@ -1348,12 +1355,12 @@
-
Now you know who has the most No. 1 hits since 2019 (as of this writing). “Are you not entertained?”
+Now you know who has the most No. 1 hits since 2020 (as of this writing). “Are you not entertained?”
4.9.1 Data Takeaway: No. 1 in past five years
OK, here is the takeaway to add …
@@ -1503,21 +1510,22 @@
Open your index.qmd file.
After the text that is here, add the lede, source graf and data takeaways noted below.
+Read through the text and many necesarry updates based on your newer data!
Then add your own data takeaways at the end.
## Summary
-Taylor Swift has had more solo appearances on the Billboard Hot 100 chart than any other artist, according to an analysis of chart data from Billboard Magazine.
+Taylor Swift has had more solo appearances on the Billboard Hot 100 chart than any other artist, according to an analysis of chart data from Billboard Magazine. Her 1,600 solo appearances tops Drake, though he has more when considering collaborations with other artists.
-The chart has been published weekly since August 1958, and this analysis includes data through December 2023*. The chart data was pulled from an [archive managed by Christian McDonald](https://github.com/utdata/rwd-billboard-data), which is updated each week.
+The chart has been published weekly since August 1958, and this analysis includes data through December 2024*. The chart data was pulled from an [archive managed by Christian McDonald](https://github.com/utdata/rwd-billboard-data), which is updated each week.
> *You should update this value with that last chart date in your data.
Other findings from the data analysis include:
- Data Takeaway: The song "Heat Wave" by Glass Animals has appeared 91 times on the Hot 100, more than any other song through 2023. The Weeknd's "Blinding Lights" has the next most appearances with 90.
-- The song "Old Town Road" by Lil Nas X Featuring Billy Ray Cyrus has spent more time at the top of the Hot 100 than any other song in history at 19 weeks.
-- The Beatles have the most No. 1 hits with 19, but they also have a 20th top song -- "Get Back" -- that is a collaboration with Billy Preston. Most other artists with double-digit top hits aren't making new music, with the exception of Taylor Swift, who has 10.
+- The songs "Old Town Road" by Lil Nas X Featuring Billy Ray Cyrus and "A Bar Song (Tipsy)" by Shaboozey have spent more time at the top of the Hot 100 than any other songs in history at 19 weeks.
+- The Beatles have the most No. 1 hits with 19, but they also have a 20th top song -- "Get Back" -- that is a collaboration with Billy Preston. Most other artists with double-digit top hits aren't making new music, with the exception of Mariah Carey and Taylor Swift.
- Taylor Swift not only has the most appearances of all time in the Hot 100, she also has the most No. 1 hits in the past five years with six.
- ADD YOUR TOP 10 HITS
- ADD YOUR OWN FINDING
@@ -1549,10 +1557,10 @@ dplyr package. Mostly we filtered and summarized our data. Here are the functions we introduced in this chapter, many with links to documentation:
filter() returns only rows that meet logical criteria you specify.str_detect() lets you search for a pattern within a string. Often used with filter().summarize() builds a summary table about your data. You can count rows n() or do math on numerical values, like mean(). In the next chapter we will summarize with math functions.group_by() is often used with summarize() to put data into groups before building a summary table based on the groups.distinct() returns rows based on unique values in columns you specify. i.e., it deduplicates data.str_detect() to search within strings.distinct() returns rows based on unique values in columns you specify. i.e., it can deduplicate data. It is NOT the same as select().
diff --git a/docs/billboard-cleaning.html b/docs/billboard-cleaning.html
index fed1c2f..d5a72d1 100644
--- a/docs/billboard-cleaning.html
+++ b/docs/billboard-cleaning.html
@@ -310,44 +310,50 @@
@@ -367,52 +373,52 @@ Table of contents
3.3 Create a new project
- 3.4 Create our cleaning notebook
- 3.5 Update our _quarto.yml
+ 3.4 Create our cleaning notebook
- 3.6 The setup chunk
+ 3.5 The setup chunk
- 3.7 About the Billboard Hot 100
+ 3.6 About the Billboard Hot 100
- 3.8 Import the data
+ 3.7 Import the data
- 3.9 Cleaning data
+ 3.8 Cleaning data
- 3.10 Exporting data
+ 3.9 Exporting data
- 3.11 Outline and bookmarks
- 3.12 Render and publish
+ 3.10 Outline and bookmarks
+ 3.11 Render and publish
- 3.13 Review of what we’ve learned so far
- 3.14 What’s next
+ 3.12 Review of what we’ve learned so far
+ 3.13 What’s next
@@ -473,7 +479,7 @@ Launch RStudio if you haven’t already.
Make sure you don’t have an existing project open. Use File > Close project if you do.
Use the +R button to create a project in a New Directory. For project type, choose Quarto Website.
-Name the project yourfirstname-billboard and put it in your ~/Documents/rwd folder.
+Name the project yourfirstname-billboard and put it in your rwd/ folder.
3.3.1 Update index with project description
@@ -505,8 +511,61 @@
-3.3.2 Create directories for your data
+
+3.3.2 Update our _quarto.yml
+The default _quarto.yml configuration file is fine, but there are some adjustments we can make to make our lives easier and our rendered project more beautiful.
+
+- We’ll add a “render” section that helps specify the order that files are processed. You’ll soon see we will name our files with numbers like
01- to specify this order. We sometime perform functions in certain notebooks, liked downloading data, that needs to be processed before other files.
+- For the
title: change the value to "Billboard Hot 100". This is the main website name that displays in the navigation bar.
+- We’ll change the format of our website navigation to a “sidebar” instead of a “navbar” that runs across the top of the page.
+- Sidebars define the content using “content” instead of “left”. We’ll set ours to list every
.qmd page we create instead of listing them individually.
+- We are adding a setting to make our rendered data frames prettier with
df-print: paged.
+- We add one more setting
code-overflow: wrap that improves the display of our code on rendered pages.
+
+Before you replace the contents, look at what you have in your _quarto.yml file compared to this below so you can recognized the differences.
+
+- Replace the contents of your
_quarto.yml file with the contents shown below.
+- Render your site again so you can see the changes.
+
+
+
+_quarto.yml
+
+project:
+ type: website
+ render:
+ - "01*.qmd"
+ - "*.qmd"
+
+website:
+ title: "Billboard Hot 100"
+ sidebar:
+ contents:
+ - href: index.qmd
+ text: Home
+ - auto: "*.qmd"
+
+format:
+ html:
+ theme: cosmo
+ css: styles.css
+ toc: true
+ df-print: paged
+ code-overflow: wrap
+
+
+
+
+3.3.3 Create directories for your data
Next we are going to create some folders in our project. We’ll use some standard names for consistency and we’ll do this for every project we build.
The first folder is called data-raw. We are creating this folder because we want to keep a pristine version of our original data that we never change or overwrite. This is a core data journalism commandment: Thou shalt not change original data.
In your Files pane at the bottom-right of RStudio, there is a New Folder icon.
@@ -528,6 +587,19 @@
+
+
+
+Project primer
+
+
+
+Since you’ll be creating projects multiple times, I’ve saved the basic steps – especially the _quarto.yml contents – in an appendix chapter: Project setup. You can use it as a reference going forward.
+
+
@@ -560,92 +632,30 @@
-3.5 Update our _quarto.yml
-
We’ve created our cleaning notebook and it will render, but it won’t show up in our website navigation yet until we specifically add it. The _quarto.yml config file controls various output options for the “website” that you are creating.
-
-- Open the
_quarto.yml file.
-- On the line that now says
- about.qmd, replace that text with - 01-cleaning.qmd.
-
-Let’s talk a little about the part of the _quarto.yml file we edited:
-website:
- title: "christian-billboard"
- navbar:
- left:
- - href: index.qmd
- text: Home
- - 01-cleaning.qmd
-Note the indents and dashes and such are important in YAML files. Under the “parent” line website we have two children, title and navbar. The navbar line has a child of left and that has two children - href and - 01-cleaning.qmd.
-When we added the 01-cleaning.qmd file it automatically added that to the nav, and then used the title of the page as word linked in the navigation.
-But for the index, we don’t want to use the title of the page. We have to define what we are doing with more detail, hence the href (meaning which page will be linked to) and the text to show in the navigation, Home.
-While we are here, we should adjust some other configurations:
-
-- For the
title: change the value to "Billboard Hot 100". This is the main website name that displays in the navigation bar.
-- At the bottom of the file on a new line, by itself and starting flush left, add
df-print: paged. This makes data we output in our notebooks look more pretty.
-
-Your file should look like this now:
-
-
-_quarto.yml
-
-project:
- type: website
-
-website:
- title: "Billboard Hot 100"
- navbar:
- left:
- - href: index.qmd
- text: Home
- - 01-cleaning.qmd
-
-format:
- html:
- theme: cosmo
- css: styles.css
- toc: true
-
-df-print: paged
-
-
-- Make sure your
_quarto.yml file is saved.
-- Go back to your Cleaning notebook and Render it to make there are no errors and that you can see the page in the website navigation.
-
-
-
-3.5.1 Describe the goals of the notebook
+
+3.4.1 Describe the goals of the notebook
At the top of the cleaning file after the YAML metadata we’ll want to explain the goals of the notebook and what we are doing.
- Add this text to your notebook AFTER the metadata:
-## Goals of this notebook
-
-The steps we'll take to prepare our data:
-
-- Download the data
-- Import it into our notebook
-- Clean up data types and columns
-- Export the data for next notebook
+## Goals of this notebook
+
+The steps we'll take to prepare our data:
+
+- Download the data
+- Import it into our notebook
+- Clean up data types and columns
+- Export the data for next notebook
We want to start each notebook with a list like this so our future selves and others know what we are trying to accomplish. It’s not unusual to update this list as we work through the notebook.
We will also write Markdown like this to explain each new “section” or goal as we tackle them.
-
-3.6 The setup chunk
+
+3.5 The setup chunk
In our previous chapter we installed several R packages onto our computer that we’ll use in almost every lesson. There are others we’ll install and use later.
But to use these packages in our notebook (and the functions they provide us) we have to load them using the library() function. We always have to declare these and convention dictates we put it near the top of a notebook so everyone understands what is needed to run our code.
-
-3.6.1 Load the libraries
+
+3.5.1 Load the libraries
- In your notebook after the goals listing, create a new “section” by adding a headline called “Setup” using Markdown, like this:
## Setup.
- After the. headline use Cmd+option+i to insert an R code chunk.
@@ -661,25 +671,25 @@
Here is the code with the chunk bits, etc:
-```{r}
-#| label: setup
-#| message: false
-
-library(tidyverse)
-library(janitor)
-```
+```{r}
+#| label: setup
+#| message: false
+
+library(tidyverse)
+library(janitor)
+```
-
-3.6.2 About the libraries
+
+3.5.2 About the libraries
A little more about the two packages we are loading here. We use them a lot.
- The
tidyverse package is actually a collection of packages for data science that are designed to work together. You can see the first time I run the chunk in the gif above that a bunch of libraries were loaded. By loading the whole tidyverse library we get readr functions for importing data, dplyr to manipulate data, lubridate to help work with dates, and ggplot to visualize data. There are more.
- The
janitor package is not maintained by Posit but it has many useful functions to clean and view data. I always include it. We’ll use janitor’s clean_names() later.
-
-3.6.3 About the options
+
+3.5.3 About the options
After adding the libraries I go back and add some lines at the top of the chunk called execution options. Let’s talk about each line:
- We start with
#| label: setup which is an execution option to “name” the code chunk. Labeling chunks is optional but useful.
@@ -704,12 +714,12 @@
-
-3.7 About the Billboard Hot 100
+
+3.6 About the Billboard Hot 100
The Billboard Hot 100 singles charts has been the music industry’s standard record chart since its inception on Aug. 4th, 1958. The rankings, published by Billboard Magazine, are currently based on sales (physical and digital), radio play, and online streaming. The methods and policies of the chart have changed over time.
The data we will use was compiled by Prof. McDonald from a number of sources. When you write about this data (and you will), you should source it as the Billboard Hot 100 from Billboard Magazine, since that is where it originally came from and they are the “owner” of the data.
-
-3.7.1 Data dictionary
+
+3.6.1 Data dictionary
Take a look at the current chart. Our data contains many (but not quite all) of the elements you see there. Each row of data (or observation as they are known in R) represents a song and the corresponding position on that week’s chart. Included in each row are the following columns (a.k.a. variables):
- CHART DATE: The release date of the chart
@@ -721,8 +731,8 @@ WKS ON CHART: The number of weeks the song has appeared as of the chart date
-
-3.7.2 Let’s download our data
+
+3.6.2 Let’s download our data
Our data is stored on a code sharing website called Github, and it is formatted as a “comma-separated value” file, or .csv. That means it basically a text file where every line is a new row of data, and each field is separated by a comma.
Because it is on the internet and has a URL, we could import it directly into our project from there, but there are benefits to getting the file onto your computer first. The hosted file could change later in ways you can’t control, and you would need Internet access to the file the next time you ran your notebook. If you download the file to keep your own copy, you have more control of your future.
Since this is a new “section” of our cleaning notebook, we’ll note what we are doing and why in Markdown.
@@ -731,11 +741,11 @@ Create an R chunk and copy/paste the following inside it. (Given the long URL, go ahead and use the copy icon at the top right of the chunk):
-download.file(
- "https://github.com/utdata/rwd-billboard-data/blob/main/data-out/hot100_assignment.csv?raw=true",
- "data-raw/hot100_assignment.csv",
- mode = "wb"
-)
+download.file(
+ "https://github.com/utdata/rwd-billboard-data/blob/main/data-out/hot100_assignment.csv?raw=true",
+ "data-raw/hot100_assignment.csv",
+ mode = "wb"
+)
Let’s explain about this:
This download.file() code is a function, or a bit of code that has been written to do a specific task. This one (surprise!) downloads files from the internet. As a function, it has some “arguments”, which are expected pieces of information that the function needs to do its job. Sometimes we just enter the arguments in the order that the function expects them, like we do here. Other times we will give it the kind of argument like mode = "w". You can search for documentation about functions in the Help tab in the bottom-right pane of RStudio.
@@ -746,10 +756,10 @@ The third argument mode = "w" gets into the weeds a little, but it helps Windows computers understand the file.
When you run the function you get an output similar to this:
-trying URL 'https://github.com/utdata/rwd-billboard-data/blob/main/data-out/hot100_assignment_current.csv?raw=true'
-Content type 'text/plain; charset=utf-8' length 18663203 bytes (17.8 MB)
+trying URL 'https://github.com/utdata/rwd-billboard-data/blob/main/data-out/hot100_assignment.csv?raw=true'
+Content type 'text/plain; charset=utf-8' length 18790353 bytes (17.9 MB)
==================================================
-downloaded 17.8 MB
+downloaded 17.9 MB
That’s not a small file at 17 MB and 300,000 rows, but it’s not a huge one, either.
You can check that this worked by going to your Files tab and clicking on the data-raw folder to go inside it and see if the file is there. To return out of the folder, click on the two dots to the right of the green up arrow.
@@ -759,8 +769,8 @@
-
3.7.3 Comment the download code
+3.6.3 Comment the download code
-
+
@@ -861,19 +871,19 @@
-
-
+- Edit your existing code chunk to look like below. You can add the
-
diff --git a/docs/explore.html b/docs/explore.html
index cd805fd..430ab8e 100644
--- a/docs/explore.html
+++ b/docs/explore.html
@@ -7,7 +7,7 @@
-Appendix E — Exploring a new dataset – Reporting with Data in R
+Appendix F — Exploring a new dataset – Reporting with Data in R
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
3.8 Import the data
+3.7 Import the data
Now that we have the data on our computer, let’s import it into our notebook so we can see it.
Since we are doing a new thing, we should again note what we are doing.
@@ -811,7 +821,7 @@- Add the following code into your chunk and run it.
-I will use the term tibble and data frame interchangeably. Think of tibbles and data frames like a well-structured spreadsheet. They are organized rows of data (called observations) with columns (called variables) where every column is a specific data type.
3.8.1 Assign our import to an R object
+3.7.1 Assign our import to an R object
Now that we know how to find our data, we next need to assign it to an R object so it can be a named thing we can reuse. We don’t want to re-import the data over and over each time we need it.
The syntax to create an object in R can seem weird at first, but the convention is to name the object first, then insert stuff into it. So, to create an object, the structure is this:
-Think of it like this: You must have a bucket before you can fill it with water. We “name” the bucket, then fill it with data. That bucket is then saved into our “environment”, meaning it is in memory where we can access it easily by calling its name.
Let’s make a object called
hot100_rawand fill it with our imported tibble.<-by using Option+- as in holding down the Option key and then pressing the hyphen:Run that chunk and several things happen:
@@ -886,23 +896,23 @@
-
+
-
-
+- After your last code chunk, add a sentence that says you will print a “glimpse” of your data.
@@ -910,10 +920,10 @@
-
-
-
-
+
+
3.8.2 Print a peek to your R Notebook
+3.7.2 Print a peek to your R Notebook
Since we can’t see the data after we assign it, let’s print the object to our notebook so we can refer to it.
- Edit your import chunk to add the last two lines of this, including
-#:3.8.3 Glimpse the data
+3.7.3 Glimpse the data
There is another way to peek at the data that I use a lot because it is more compact and shows you all the columns and data examples without scrolling:
glimpse().I’m showing the return here as well. Afterward I’ll explain the pipe:
|>.
diff --git a/docs/denied-cleaning.html b/docs/denied-cleaning.html index 71b49fd..038a0e1 100644 --- a/docs/denied-cleaning.html +++ b/docs/denied-cleaning.html @@ -310,44 +310,50 @@-
+
+ A Project setup
+
+
+ -
+
- A Using posit.cloud
+ B Using posit.cloud
-
- B R Functions
+ C R Functions
-
- C Troubleshooting
+ D Troubleshooting
-
- D Asking for help
+ E Asking for help
-
- E Exploring a new dataset
+ F Exploring a new dataset
-
- F Chart production tips
+ G Chart production tips
-
- G Resources
+ H Resources
@@ -1069,7 +1075,7 @@Appendix A — Project setup
+When you create a new “Quarto Website” it creates the shell of a website with some starter pages.
+For this class, you should update your project in the following ways …
+A.1 Update your quarto file
+Replace your
+_quarto.ymlfile with the contents below. Update thetitleas appropriate.A.2 Data folders
+Create two folders to hold your data
++
+data-rawis where you store original data that you don’t want to edit or overwrite.data-processedis where you save any new outputs from your notebook.A.3 Quarto file names
+Most projects will have at least three files:
++
+index.qmdis the home page of your site. It should have an explanation of the project and perhaps a summary of what you’ve learned.01-cleaning.qmdis where you prepare data for analysis. Cleaned files should be output from this file as .rds files intodata-processed.02-analysis.qmdis where your analysis is done. It should run independently, meaning you should start by importing the cleaned data.You can delete the
+ + +about.qmdpage that is created with the default project.