Securely storing API keys

Tutorial
Application programming interface
What are API keys and how to store them securely for R packages.
Modified

March 11, 2024

library(tidyverse)
library(wordcloud)
library(tidytext)
library(scales)
library(viridis)

# set default seed and theme
set.seed(123)
theme_set(theme_minimal())

What are API keys?

An application programming interface (API) key is a unique identifier used to authenticate and authorize a user, developer, or calling program to an API.1 API keys are used to track and control how the API is being used, for example to prevent malicious use or abuse of the API. The API key often acts as both a unique identifier and a secret token for authentication, and is assigned a set of access that is specific to the identity that is associated with it.

Depending on how the API is set up, you may need to include your API key in every request you make, or you may only need to include it once to get a token that you can use for subsequent requests. In either case, you should never share your API key with anyone else. If you do, they will be able to use the API as if they were you, and you may be held responsible for any misuse of the API.

Searching geographic info: geonames

library(geonames)

API authentication

There are a few things we need to do to be able to use this package to access the geonames API:

  1. Go to the geonames site and register an account.
  2. Click here to enable the free web service
  3. Tell R your geonames username. You could run the line
options(geonamesUsername = "<YOUR USER NAME>")

in R. However this is insecure. We don’t want to risk committing this line and pushing it to our public GitHub page! Instead, you should create a file in the same place as your .Rproj file. To do that, run the following command from the R console:

usethis::edit_r_profile(scope = "project")

This will create a special file called .Rprofile in the same directory as your .Rproj file (assuming you are working in an R project). The file should open automatically in your RStudio script editor. Add

options(geonamesUsername = "<YOUR USER NAME>")

to that file, replacing <YOUR USER NAME> with your Geonames username.

Important

  • Make sure your .Rprofile ends with a blank line
  • Make sure .Rprofile is included in your .gitignore file, otherwise it will be synced with Github
  • Restart RStudio after modifying .Rprofile in order to load any new keys into memory
  • Spelling is important when you set the option in your .Rprofile
  • You can do a similar process for an arbitrary package or key. For example:
# in .Rprofile
options(this_is_my_key = "XXXX")

# later, in the R script:
key <- getOption("this_is_my_key")

This is a simple means to keep your keys private, especially if you are sharing the same authentication across several projects. Remember that using .Rprofile makes your code un-reproducible. In this case, that is exactly what we want!

Using Geonames

What can we do? Get access to lots of geographical information via the various “web services”

countryInfo <- GNcountryInfo()
countryInfo |>
  as_tibble() |>
  glimpse()
Rows: 250
Columns: 18
$ continent        <chr> "EU", "AS", "AS", "NA", "NA", "EU", "AS", "AF", "AN",…
$ capital          <chr> "Andorra la Vella", "Abu Dhabi", "Kabul", "Saint John…
$ languages        <chr> "ca", "ar-AE,fa,en,hi,ur", "fa-AF,ps,uz-AF,tk", "en-A…
$ geonameId        <chr> "3041565", "290557", "1149361", "3576396", "3573511",…
$ south            <chr> "42.4287475", "22.6315119400001", "29.3770645357176",…
$ isoAlpha3        <chr> "AND", "ARE", "AFG", "ATG", "AIA", "ALB", "ARM", "AGO…
$ north            <chr> "42.6558875", "26.0693916590001", "38.4907920755748",…
$ fipsCode         <chr> "AN", "AE", "AF", "AC", "AV", "AL", "AM", "AO", "AY",…
$ population       <chr> "77006", "9630959", "37172386", "96286", "13254", "28…
$ east             <chr> "1.7866939", "56.381222289", "74.8894511481168", "-61…
$ isoNumeric       <chr> "020", "784", "004", "028", "660", "008", "051", "024…
$ areaInSqKm       <chr> "468.0", "82880.0", "647500.0", "443.0", "102.0", "28…
$ countryCode      <chr> "AD", "AE", "AF", "AG", "AI", "AL", "AM", "AO", "AQ",…
$ west             <chr> "1.4135734", "51.5904085340001", "60.4720833972263", …
$ countryName      <chr> "Principality of Andorra", "United Arab Emirates", "I…
$ postalCodeFormat <chr> "AD###", "", "", "", "", "####", "######", "", "", "@…
$ continentName    <chr> "Europe", "Asia", "Asia", "North America", "North Ame…
$ currencyCode     <chr> "EUR", "AED", "AFN", "XCD", "XCD", "ALL", "AMD", "AOA…

This country info dataset is very helpful for accessing the rest of the data, because it gives us the standardized codes for country and language.

The Manifesto Project: manifestoR

The Manifesto Project collects and organizes political party manifestos from around the world. It currently covers over 1000 parties from 1945 until today in over 50 countries on five continents. We can use the manifestoR package to access the API and download those manifestos for analysis in R.

Load library and set API key

Accessing data from the Manifesto Project API requires an authentication key. You can create an account and key here. Here I store my key in .Rprofile and retrieve it using mp_setapikey().

library(manifestoR)

# retrieve API key stored in .Rprofile
mp_setapikey(key = getOption("manifesto_key"))

Once you have done this step, your API key is automatically included in every request you make to the Manifesto Project API.

Retrieve the database

mpds <- mp_maindataset()
Connecting to Manifesto Project DB API... 
Connecting to Manifesto Project DB API... corpus version: 2023-1 
mpds
# A tibble: 5,089 × 175
   country countryname oecdmember eumember edate        date party partyname    
     <dbl> <chr>            <dbl>    <dbl> <date>      <dbl> <dbl> <chr>        
 1      11 Sweden               0        0 1944-09-17 194409 11220 Communist Pa…
 2      11 Sweden               0        0 1944-09-17 194409 11320 Social Democ…
 3      11 Sweden               0        0 1944-09-17 194409 11420 People’s Par…
 4      11 Sweden               0        0 1944-09-17 194409 11620 Right Party  
 5      11 Sweden               0        0 1944-09-17 194409 11810 Agrarian Par…
 6      11 Sweden               0        0 1948-09-19 194809 11220 Communist Pa…
 7      11 Sweden               0        0 1948-09-19 194809 11320 Social Democ…
 8      11 Sweden               0        0 1948-09-19 194809 11420 People’s Par…
 9      11 Sweden               0        0 1948-09-19 194809 11620 Right Party  
10      11 Sweden               0        0 1948-09-19 194809 11810 Agrarian Par…
# ℹ 5,079 more rows
# ℹ 167 more variables: partyabbrev <chr>, parfam <dbl>, candidatename <chr>,
#   coderid <dbl>, manual <dbl>, coderyear <dbl>, testresult <dbl>,
#   testeditsim <dbl>, pervote <dbl>, voteest <dbl>, presvote <dbl>,
#   absseat <dbl>, totseats <dbl>, progtype <dbl>, datasetorigin <dbl>,
#   corpusversion <chr>, total <dbl>, peruncod <dbl>, per101 <dbl>,
#   per102 <dbl>, per103 <dbl>, per104 <dbl>, per105 <dbl>, per106 <dbl>, …

mp_maindataset() includes a data frame describing each manifesto included in the database. You can use this database for some exploratory data analysis. For instance, how many manifestos have been published by each political party in Sweden?

mpds |>
  filter(countryname == "Sweden") |>
  count(partyname) |>
  mutate(partyname = fct_reorder(.f = partyname, .x = n)) |>
  ggplot(mapping = aes(x = n, y = partyname)) +
  geom_col() +
  labs(
    title = "Political manifestos published in Sweden",
    x = NULL,
    y = "Total (1948-present)"
  )

Or we can use scaling functions to identify each party manifesto on an ideological dimension. For example, how have the Democratic and Republican Party manifestos in the United States changed over time?

mpds_usa <- mpds |>
  filter(party == 61320 | party == 61620) 

mpds_usa |>
  mutate(ideo = mp_scale(mpds_usa)) |>
  select(partyname, edate, ideo) |>
  ggplot(aes(edate, ideo, color = partyname)) +
  geom_line() +
  scale_color_manual(values = c("blue", "red")) +
  labs(
    title = "Ideological scaling of major US political parties",
    x = "Year",
    y = "Ideological position",
    color = NULL
  ) +
  theme(legend.position = "bottom")

Download manifestos

mp_corpus() can be used to download the original manifestos as full text documents stored as a corpus. Once you obtain the corpus, you can perform text analysis. As an example, let’s compare the most common words in the Democratic and Republican Party manifestos from the 2020 U.S. presidential election:

# download documents
docs <- mp_corpus(countryname == "United States" & edate > as.Date("2020-01-01"))
Connecting to Manifesto Project DB API... 
Connecting to Manifesto Project DB API... corpus version: 2023-1 
Connecting to Manifesto Project DB API... corpus version: 2023-1 
Connecting to Manifesto Project DB API... corpus version: 2023-1 
docs
<<ManifestoCorpus>>
Metadata:  corpus specific: 0, document level (indexed): 0
Content:  documents: 2
# generate wordcloud of most common terms
docs |>
  tidy() |>
  mutate(party = factor(x = party,
    levels = c(61320, 61620),
    labels = c("Democratic Party", "Republican Party")
  )) |>
  unnest_tokens(word, text) |>
  anti_join(stop_words) |>
  count(party, word, sort = TRUE) |>
  drop_na() |>
  reshape2::acast(word ~ party, value.var = "n", fill = 0) |>
  comparison.cloud(max.words = 200)

Census data with tidycensus

tidycensus provides an interface with the US Census Bureau’s decennial census and American Community APIs and returns tidy data frames with optional simple feature geometry. These APIs require a free key you can obtain here.

Rather than storing your key in .Rprofile, tidycensus includes census_api_key() which automatically stores your key in .Renviron, which is a location to store environment variables. Anything stored in .Renviron is automatically loaded anytime you initiate R on your computer, regardless of the project or file location. Once you get your key, load it:

library(tidycensus)
census_api_key("YOUR API KEY GOES HERE", install = TRUE)
Tip

All future requests to the Census API using tidycensus on your computer will automatically include your key. You will not need to run census_api_key() again.

Obtaining decennial census data

get_decennial() allows you to obtain data from the 1990, 2000, 2010, and 2020 decennial US censuses. Let’s look at the number of individuals of Asian ethnicity by state in 2020:2

asia20 <- get_decennial(geography = "state", variables = "P1_006N", year = 2020)
Getting data from the 2020 decennial Census
Using the PL 94-171 Redistricting Data Summary File
Note: 2020 decennial Census data use differential privacy, a technique that
introduces errors into data to preserve respondent confidentiality.
ℹ Small counts should be interpreted with caution.
ℹ See https://www.census.gov/library/fact-sheets/2021/protecting-the-confidentiality-of-the-2020-census-redistricting-data.html for additional guidance.
This message is displayed once per session.
asia20
# A tibble: 52 × 4
   GEOID NAME                 variable   value
   <chr> <chr>                <chr>      <dbl>
 1 42    Pennsylvania         P1_006N   510501
 2 06    California           P1_006N  6085947
 3 54    West Virginia        P1_006N    15109
 4 49    Utah                 P1_006N    80438
 5 36    New York             P1_006N  1933127
 6 11    District of Columbia P1_006N    33545
 7 02    Alaska               P1_006N    44032
 8 12    Florida              P1_006N   643682
 9 45    South Carolina       P1_006N    90466
10 38    North Dakota         P1_006N    13213
# ℹ 42 more rows

The result of get_decennial() is a tidy data frame with one row per geographic unit-variable.

  • GEOID - identifier for the geographical unit associated with the row
  • NAME - descriptive name of the geographical unit
  • variable - the Census variable encoded in the row
  • value - the value of the variable for that geographic unit

We can quickly visualize this data frame using ggplot2:

asia20 |>
  mutate(NAME = fct_reorder(.f = NAME, .x = value)) |>
  ggplot(mapping = aes(x = value, y = NAME)) +
  geom_point() +
  scale_x_continuous(labels = label_comma(scale_cut = cut_short_scale())) +
  labs(
    x = NULL,
    y = "Number of residents of Asian ethnicity"
  )

Of course this graph is not entirely useful since it is based on the raw frequency of Asian individuals. California is at the top of the list, but it is also the most populous state. Instead, we could normalize this value as a percentage of the entire state population. To do that, we need to retrieve another variable:

asia_pop <- get_decennial(
  geography = "state",
  variables = c(asian = "P1_006N", total = "P1_001N"),
  year = 2020,
  output = "wide"
)
Getting data from the 2020 decennial Census
Using the PL 94-171 Redistricting Data Summary File
asia_pop
# A tibble: 52 × 4
   GEOID NAME                   asian    total
   <chr> <chr>                  <dbl>    <dbl>
 1 42    Pennsylvania          510501 13002700
 2 06    California           6085947 39538223
 3 54    West Virginia          15109  1793716
 4 49    Utah                   80438  3271616
 5 36    New York             1933127 20201249
 6 11    District of Columbia   33545   689545
 7 02    Alaska                 44032   733391
 8 12    Florida               643682 21538187
 9 45    South Carolina         90466  5118425
10 38    North Dakota           13213   779094
# ℹ 42 more rows
asia_pop |>
  mutate(asian_pct = asian / total,
         NAME = fct_reorder(.f = NAME, .x = asian_pct)) |>
  ggplot(mapping = aes(x = asian_pct, y = NAME)) +
  geom_point() +
  scale_x_continuous(labels = label_percent()) +
  labs(
    x = NULL,
    y = "Percent of residents of Asian ethnicity"
  )

Obtaining American Community Survey data

get_acs() retrieves data from the American Community Survey. This survey is administered to a sample of 3 million households on an annual basis, so the data points are estimates characterized by a margin of error. tidycensus returns both the original estimate and margin of error. Let’s get median household income data from the 2018-2012 ACS for each state.

usa_inc <- get_acs(
  geography = "state",
  variables = c(medincome = "B19013_001"),
  year = 2022
)
Getting data from the 2018-2022 5-year ACS
usa_inc
# A tibble: 52 × 5
   GEOID NAME                 variable  estimate   moe
   <chr> <chr>                <chr>        <dbl> <dbl>
 1 01    Alabama              medincome    59609   377
 2 02    Alaska               medincome    86370  1083
 3 04    Arizona              medincome    72581   450
 4 05    Arkansas             medincome    56335   422
 5 06    California           medincome    91905   277
 6 08    Colorado             medincome    87598   508
 7 09    Connecticut          medincome    90213   730
 8 10    Delaware             medincome    79325  1227
 9 11    District of Columbia medincome   101722  1569
10 12    Florida              medincome    67917   259
# ℹ 42 more rows

Now we return both an estimate column for the ACS estimate and moe for the margin of error (defaults to 90% confidence interval).

usa_inc |>
  mutate(NAME = fct_reorder(.f = NAME, .x = estimate)) |>
  ggplot(mapping = aes(x = estimate, y = NAME)) +
  geom_pointrange(mapping = aes(
    xmin = estimate - moe,
    xmax = estimate + moe
  ),
  size = .25
  ) +
  scale_x_continuous(labels = label_dollar()) +
  labs(
    title = "Household income by state",
    subtitle = "2022 American Community Survey (five-year estimates)",
    x = NULL,
    y = "ACS estimate (bars represent margin of error)"
  )

Search for variables

get_acs() or get_decennial() requires knowing the variable ID, of which there are thousands. load_variables() downloads a list of variable IDs and labels for a given Census or ACS and dataset. You can then use view() to interactively browse through and filter for variables in RStudio.

Drawing maps

tidycensus also can return simple feature geometry for geographic units along with variables from the decennial Census or ACS, which can then be visualized using geom_sf(). Let’s look at median household income by Census tracts from the 2018-2022 ACS in Tompkins County, NY:

tompkins <- get_acs(
  state = "NY",
  county = "Tompkins",
  geography = "tract",
  variables = c(medincome = "B19013_001"),
  year = 2022,
  geometry = TRUE
)
tompkins
Simple feature collection with 26 features and 5 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -76.69666 ymin: 42.26298 xmax: -76.23782 ymax: 42.62742
Geodetic CRS:  NAD83
First 10 features:
         GEOID                                         NAME  variable estimate
1  36109000300    Census Tract 3; Tompkins County; New York medincome       NA
2  36109000800    Census Tract 8; Tompkins County; New York medincome    52188
3  36109000100    Census Tract 1; Tompkins County; New York medincome    39512
4  36109000600    Census Tract 6; Tompkins County; New York medincome    97500
5  36109000201 Census Tract 2.01; Tompkins County; New York medincome       NA
6  36109000900    Census Tract 9; Tompkins County; New York medincome    83510
7  36109000400    Census Tract 4; Tompkins County; New York medincome    53750
8  36109002300   Census Tract 23; Tompkins County; New York medincome    85295
9  36109000700    Census Tract 7; Tompkins County; New York medincome    56578
10 36109001500   Census Tract 15; Tompkins County; New York medincome    76841
     moe                       geometry
1     NA MULTIPOLYGON (((-76.48981 4...
2  24145 MULTIPOLYGON (((-76.51474 4...
3   5060 MULTIPOLYGON (((-76.50839 4...
4  44884 MULTIPOLYGON (((-76.52701 4...
5     NA MULTIPOLYGON (((-76.48959 4...
6  11361 MULTIPOLYGON (((-76.57343 4...
7  27113 MULTIPOLYGON (((-76.48973 4...
8  16628 MULTIPOLYGON (((-76.66654 4...
9  10038 MULTIPOLYGON (((-76.51177 4...
10 20642 MULTIPOLYGON (((-76.53789 4...

This looks similar to the previous output but because we set geometry = TRUE it is now a simple features data frame with a geometry column defining the geographic feature. We can visualize it using geom_sf() and viridis::scale_*_viridis() to adjust the color palette.

ggplot(data = tompkins) +
  geom_sf(mapping = aes(fill = estimate, color = estimate)) +
  coord_sf(crs = 26911) +
  scale_fill_viridis_c(
    option = "magma",
    labels = label_dollar(),
    aesthetics = c("fill", "color")
  ) +
  labs(
    title = "Household income by census tract",
    subtitle = "Tompkins County, NY",
    caption = "Source: 2022 American Community Survey (five-year estimates)",
    fill = NULL,
    color = NULL
  )

Acknowledgments

sessioninfo::session_info()
─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.2 (2023-10-31)
 os       macOS Ventura 13.5.2
 system   aarch64, darwin20
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/New_York
 date     2024-03-11
 pandoc   3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 base64enc     0.1-3   2015-07-28 [1] CRAN (R 4.3.0)
 cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.1)
 digest        0.6.34  2024-01-11 [1] CRAN (R 4.3.1)
 dplyr         1.1.4   2023-11-17 [1] CRAN (R 4.3.1)
 DT            0.31    2023-12-09 [1] CRAN (R 4.3.1)
 evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.1)
 fansi         1.0.6   2023-12-08 [1] CRAN (R 4.3.1)
 fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
 functional    0.6     2014-07-16 [1] CRAN (R 4.3.0)
 generics      0.1.3   2022-07-05 [1] CRAN (R 4.3.0)
 glue          1.7.0   2024-01-09 [1] CRAN (R 4.3.1)
 here          1.0.1   2020-12-13 [1] CRAN (R 4.3.0)
 hms           1.1.3   2023-03-21 [1] CRAN (R 4.3.0)
 htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.1)
 htmlwidgets   1.6.4   2023-12-06 [1] CRAN (R 4.3.1)
 jsonlite      1.8.8   2023-12-04 [1] CRAN (R 4.3.1)
 knitr         1.45    2023-10-30 [1] CRAN (R 4.3.1)
 lattice       0.21-9  2023-10-01 [1] CRAN (R 4.3.2)
 lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.1)
 magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
 manifestoR  * 1.5.0   2020-11-29 [1] CRAN (R 4.3.0)
 mnormt        2.1.1   2022-09-26 [1] CRAN (R 4.3.0)
 nlme          3.1-163 2023-08-09 [1] CRAN (R 4.3.2)
 NLP         * 0.2-1   2020-10-14 [1] CRAN (R 4.3.0)
 pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.0)
 pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.3.0)
 psych         2.3.12  2023-12-20 [1] CRAN (R 4.3.1)
 purrr         1.0.2   2023-08-10 [1] CRAN (R 4.3.0)
 R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
 Rcpp          1.0.12  2024-01-09 [1] CRAN (R 4.3.1)
 readr         2.1.5   2024-01-10 [1] CRAN (R 4.3.1)
 rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.1)
 rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.1)
 rprojroot     2.0.4   2023-11-05 [1] CRAN (R 4.3.1)
 rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.0)
 sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
 slam          0.1-50  2022-01-08 [1] CRAN (R 4.3.0)
 tibble        3.2.1   2023-03-20 [1] CRAN (R 4.3.0)
 tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.3.0)
 tm          * 0.7-11  2023-02-05 [1] CRAN (R 4.3.0)
 tzdb          0.4.0   2023-05-12 [1] CRAN (R 4.3.0)
 utf8          1.2.4   2023-10-22 [1] CRAN (R 4.3.1)
 vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.3.1)
 xfun          0.41    2023-11-01 [1] CRAN (R 4.3.1)
 xml2          1.3.6   2023-12-04 [1] CRAN (R 4.3.1)
 yaml          2.3.8   2023-12-11 [1] CRAN (R 4.3.1)
 zoo           1.8-12  2023-04-13 [1] CRAN (R 4.3.0)

 [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library

──────────────────────────────────────────────────────────────────────────────

Footnotes

  1. Source: Wikipedia↩︎

  2. Specifically all individuals who identify as Asian alone and not in combination with another race.↩︎