mothur

README for the greengenes2 2024_09 reference files

Mon, 27 Apr 2026 00:00:00 +0000

The biocore group released an updated version of the greengenes taxonomy in September 2024, which was an update of v.2022.10 that had been published in Nature Biotechnology. If you use these files, you should cite McDonald et al. My understanding is that the biggest changes you should notice will be a change in names to follow the GTDB 220.

Looking through their ftp site you will find a number of files that are useful for classifying ASVs, metagenomes, etc. For mothur-based workflows using the naive Bayesian classifier, we are interested in the full-length sequences. It is worth noting that the taxonomy provided with greengenes2 goes out to the species level. This does not mean that you should expect to be able to classify your sequences to the species level. There is general agreement that to achieve that you will need to have genome sequences.

Here is how I generated the mothur-compatible greengenes2 files.

To download the files we want two qza files (these are zip files that are used with QIIME 2). We can download them, extract them, and get them in an easy to find location using these bash commands:

wget --no-check-certificate https://ftp.microbio.me/greengenes_release/2024.09/2024.09.backbone.full-length.fna.qza
wget --no-check-certificate https://ftp.microbio.me/greengenes_release/2024.09/2024.09.backbone.tax.qza

unzip 2024.09.backbone.full-length.fna.qza
unzip 2024.09.backbone.tax.qza

mv */data/dna-sequences.fasta dna-sequences.fasta
mv */data/taxonomy.tsv taxonomy.tsv

Then in R we will read in the fasta and taxonomy files, make sure they’re in the correct order and polish the taxonomy strings to work with mothur:

library(tidyverse)
library(vroom)

fasta_fname <- "dna-sequences.fasta"
tax_fname <- "taxonomy.tsv"

f <- vroom_lines(fasta_fname)
indices <- seq_along(f)

seq_data <- tibble(id = f[indices %% 2 == 1],
      seq = f[indices %% 2 == 0]) %>%
  mutate(id = stringi::stri_replace_first_fixed(id, ">", ""))


tax_data <- vroom(tax_fname, delim = "\t",
                  col_names = c("id", "taxonomy"),
                  col_types = "cc",
                  skip = 1)


s_t <- anti_join(seq_data, tax_data, by = "id") %>% nrow(.) == 0
t_s <- anti_join(tax_data, seq_data, by = "id") %>% nrow(.) == 0
stopifnot(s_t, t_s)

seq_tax_data <- inner_join(seq_data, tax_data, by = "id")

Let’s look at the data a bit…

parsed_tax <- seq_tax_data %>%
  select(id, taxonomy) %>%
  separate(taxonomy, sep = "; ", into = c("k", "p", "c", "o", "f", "g", "s"))

count(parsed_tax, k)

There are 335,412 bacterial and 2,094 archaeal sequences. Here is the number of kingdoms through species included in the reference and the number of taxa at each level that only has one sequence.

parsed_tax %>%
  pivot_longer(-id) %>%
  filter(!str_detect(value, "__$")) %>%
  nest(data = -name) %>%
  mutate(summary = map_dfr(data, \(x){
    z <- count(x, value)
    tibble(n_taxa = nrow(z), n_singletons = sum(z$n == 1))
  })) %>%
  select(name, summary) %>%
  unnest(summary)

# A tibble: 7 × 3
  name  n_taxa n_singletons
  <chr>  <int>        <int>
k          2            0
p        132            3
c        333           21
o        957          105
f       2107          315
g       8114         2023
s      22901        10316

Here’s the number of sequences with a name at each taxonomic level

parsed_tax %>%
  pivot_longer(-id) %>%
  mutate(name = factor(name, levels = c("k", "p", "c", "o", "f", "g", "s"))) %>%
  filter(!str_detect(value, "__$")) %>%
  count(name)

# A tibble: 7 × 2
  name       n
  <fct>  <int>
k     337506
p     337431
c     336876
o     330113
f     323589
g     290541
s     199527

About 59% of sequences have a species level name. Not all sequences have genera-level names and not all genera have species-level names.

parsed_tax %>%
  select(g, s) %>%
  distinct() %>%
  summarize(n_species = sum(s != "s__"), .by = g) %>% 
  count(n_species, name = "n_genera") %>%
  select(n_genera, n_species) %>%
  print(n = Inf)

   n_genera n_species
      <int>     <int>
   5473         1
   1037         2
    483         3
    278         4
    170         5
    106         6
     89         7
     68         8
     55         9
     40        10
     42        11
     37        12
     21        13
     23        14
     17        15
      9        16
     12        17
     12        18
      7        19
      6        20
      6        21
      4        22
      9        23
      5        24
      9        25
      7        26
      8        27
      2        28
      4        29
      2        30
      4        31
      1        32
      4        33
      6        34
      1        35
      1        36
      5        37
      2        38
      1        39
      4        40
      1        41
      1        43
      2        44
      3        45
      1        46
      1        47
      3        48
      2        51
      1        52
      1        54
      1        57
      1        61
      1        63
      1        66
      1        67
      1        68
      1        70
      2        72
      1        75
      1        77
      1        79
      2        80
      1        82
      2        85
      1        90
      1       106
      1       114
      1       116
      1       119
      1       134
      1       136
      1       139
      1       166
      1       201
      1       266
      1       675

This tells us that there are 5373 genera with only one species; there’s also one genus with 675 species. This tells me that if you pick one of the 5373 singleton genera, you will definitely get that species - that’s about two-thirds of the genera. I worry that this will give the sense of an unrealistic sense of specificity for sequences. With that in mind, I will post the version of the greengenes2 database without species-level names. The code for generating the results with the species-level names is also included below in case you want to go out on that limb.

Let’s do the one with species-level names first:

seq_tax_data %>%
  select(id, taxonomy) %>%
  mutate(
    taxonomy = stringi::stri_replace_all_regex(taxonomy, " ", ""),
    taxonomy = stringi::stri_replace_all_regex(taxonomy, "$", ";")
  ) %>%
  write_tsv("greengenes2_2024_09.w_sp.taxonomy", col_names = FALSE)

seq_tax_data %>%
  select(id, seq) %>%
  mutate(id = stringi::stri_replace_first_regex(id, "^", ">")) %>%
  unite(fasta, id, seq, sep = "\n") %>%
  write_tsv("greengenes2_2024_09.w_sp.fasta", col_names = FALSE)

Let’s do the one without species-level names next:

seq_tax_data %>%
  select(id, taxonomy) %>%
  mutate(
    taxonomy = stringi::stri_replace_all_regex(taxonomy, " ", ""),
    taxonomy = stringi::stri_replace_all_regex(taxonomy, "s__.*", "")
  ) %>%
  write_tsv("greengenes2_2024_09.wo_sp.taxonomy", col_names = FALSE)

seq_tax_data %>%
  select(id, seq) %>%
  mutate(id = stringi::stri_replace_first_regex(id, "^", ">")) %>%
  unite(fasta, id, seq, sep = "\n") %>%
  write_tsv("greengenes2_2024_09.wo_sp.fasta", col_names = FALSE)

Let’s package it all together…

mkdir greengenes2_2024_09.wo_sp
mv greengenes2_2024_09.wo_sp.* greengenes2_2024_09.wo_sp
cp README.md greengenes2_2024_09.wo_sp
tar cvzf greengenes2_2024_09.wo_sp.tgz greengenes2_2024_09.wo_sp

README for the SILVA v138.2 reference files

Thu, 26 Sep 2024 00:00:00 +0000

The good people at SILVA have released a new version of the SILVA v138 (and v138.1) database. My understanding is that this update removed 13 sequences from v138. The biggest change was a number of modifications to the taxonomy including applying 6 taxonomic levels and using “Incertae Sedis” instead of “unclassified”. A little bit of tweaking is needed to get their files to be compatible with mothur. This README document describes the process that I used to generate the mothur-compatible reference files.

Curation of references

Getting the data in and out of the ARB database

This README file explains how we generated the silva reference files for use with mothur’s classify.seqs and align.seqs commands. I’ll assume that you have a functioning copy of arb installed on your computer. For this README we are using version 6.0. First we need to download the database and decompress it. From the command line we do the following:

wget -N https://www.arb-silva.de/fileadmin/silva_databases/release_138_2/ARB_files/SILVA_138.2_SSURef_NR99_03_07_24_opt.arb.gz
gunzip SILVA_138.2_SSURef_NR99_03_07_24_opt.arb.gz
arb SILVA_138.2_SSURef_NR99_03_07_24_opt.arb

This will launch us into the arb environment with the ‘‘Ref NR 99’’ database opened. This database has 510,495 sequences within it that are not more than 99% similar to each other. The release notes for this database as well as the idea behind the non-redundant database are available from the silva website. Within arb do the following:

Click the search button
Set the first search field to ‘ARB_color’ and set it to 1. Click on the equal sign until it indicates not equal (this removes low quality reads and chimeras)
Click ‘Search’. This yielded 446,875 hits
Click the “Mark Listed Unmark Rest” button
Close the “Search and Query” box
Now click on File->export->export to external format
In this box the Export option should be set to marked, Filter to none, and Compression should be set to no.
In the field for Choose an output file name make sure the path has you in the correct working directory and enter silva.full_v138_2.fasta`.
Select a format: fasta_mothur.eft. This is a custom formatting file that I have created that includes the sequences accession number and it’s taxonomy across the top line. To create one for you will need to create fasta_mothur.eft in the $ARBHOME/lib/export/ folder with the following:
```
SUFFIX          fasta
BEGIN
>*(acc).*(name)\t*(align_ident_slv)\t*(tax_slv);
*(|export_sequence)
```
Save this as silva.full_v138_2.fasta
You can now quit arb.

Screening the sequences

Now we need to screen the sequences for those that span the 27f and 1492r primer region, have 5 or fewer ambiguous base calls, and that are unique. We’ll also extract the taxonomic information from the header line. Run the following commands from a bash terminal:

#Convert from RNA to DNA sequences...
sed '/^[^>]/s/[Uu]/T/g' silva.full_v138_2.fasta > silva.full_v138_2_dna.fasta

mothur "#screen.seqs(fasta=silva.full_v138_2_dna.fasta, start=1044, end=43116, maxambig=5);
        pcr.seqs(start=1044, end=43116, keepdots=T);
        degap.seqs();
        unique.seqs();"

#identify the unique sequences without regard to their alignment
grep ">" silva.full_v138_2_dna.good.pcr.ng.unique.fasta | cut -f 1 | cut -c 2- > silva.full_v138_2_dna.good.pcr.ng.unique.accnos

#get the unique sequences without regard to their alignment
mothur "#get.seqs(fasta=silva.full_v138_2_dna.good.pcr.fasta, accnos=silva.full_v138_2_dna.good.pcr.ng.unique.accnos)"

#generate alignment file
mv silva.full_v138_2_dna.good.pcr.pick.fasta silva.nr_v138_2.align

#generate taxonomy file
grep '>' silva.nr_v138_2.align | cut -f1,3 | cut -f2 -d'>' > silva.nr_v138.full

The mothur commands above do several things. First the screen.seqs command removes sequences that are not full length and have more than 5 ambiguous base calls. Note: this will remove a number of Archaea since the ARB RN reference database lets in shorter (>900 bp) archaeal 16S rRNA gene sequences. Second, pcr.seqs converts any base calls that occur before position 1044 and after 43116 to . to make them only span the region between the 27f and 1492r priming sites. Finally, it is possible that weird things happen in the alignments and so we unalign the sequences (degap.seqs) and identify the unique sequences (unique.seqs). We then convert the resulting fasta file into an accnos file so that we can go back into mothur and pull out the unique sequences from the aligned file (get.seqs).

Formatting the taxonomy files

Now we want to make sure the taxonomy file is properly formatted for use with mothur. First we want to grab the SILVA taxa mapping file by running the following in bash:

wget https://www.arb-silva.de/fileadmin/silva_databases/release_138_2/Exports/taxonomy/tax_slv_ssu_138.2.txt.gz
gunzip tax_slv_ssu_138.2.txt.gz

We’ll run the following code from within R to clean up the taxa names and make sure everything has six levels:

library(tidyverse)

desired_levels <- c("domain", "phylum", "class", "order", "family", "genus")
desired_levels_tbl <- tibble(
  tax_level = factor(desired_levels, desired_levels))

# this is their reference taxonomy with levels for each substring found
# in the database
tax_label_level <- read_tsv("tax_slv_ssu_138.2.txt", col_names = FALSE, 
                            col_type = cols(.default = col_character())) %>%
  select(tax_label = X1, tax_level = X3)


# this is the the full taxonoy for each sequence in the database
database_tax_label <- read_tsv("silva.nr_v138.full",
                               col_names = c("id", "tax_label"),
                               col_type = cols(.default = col_character()))

# these are the unique tax_label values found in database_tax_label
unique_tax_labels <- database_tax_label %>%
  select(tax_label) %>%
  distinct() %>%
  left_join(tax_label_level, by = "tax_label") %>%
  select(tax_label)


# now need to get each of the substrings found in unique_tax_labels and return
# the tax_level for each substring taxonomy
generate_substrings <- function(s) {
  words <- str_replace(s, ";$", "") |>
    str_split(";") |>
    unlist()
  
  substrings <- character(length(words))
  for(w in seq_along(words)){
    substrings[w] <- paste(paste(words[1:w], collapse = ";"), "", sep = ";")
  }
  substrings
}

# replace missing levels with insertae sedis of the previous good name with
# the taxonomic level appended
fill_ss_tbl <- function(ss_tbl) {
  
  if(nrow(ss_tbl) != 6) {
    
    ss_tbl <- ss_tbl %>%
      right_join(desired_levels_tbl, by = "tax_level")
    
    nas <- which(is.na(ss_tbl$substring))
    previous_good_string <- ""
    
    for(n in nas){
      if(!str_detect(ss_tbl[n - 1, "substring"], "_insertae_sedis_")){
        previous_good_string <- n - 1
      }
      
      ss_tbl[n, "substring"] <- paste0(ss_tbl[previous_good_string, "substring"],
                                       "_insertae_sedis_",
                                       ss_tbl[n, "tax_level"])
    }
  }
  
  str_replace_all(paste(paste(ss_tbl$substring, collapse = ";"), "", sep = ";"),
                  " ",
                  "_")
}


clean_tax_labels_lookup <- unique_tax_labels %>%
  mutate(substring = map(tax_label, generate_substrings)) %>% #generate substrs
  unnest(substring) %>%
  inner_join(tax_label_level, by = c("substring" = "tax_label")) %>%
  mutate(substring = str_replace(substring, "^.*?([^;]+);$", "\\1")) %>%
  filter(!str_detect(substring, "^Incertae Sedis$")) %>% 
  select(tax_label, substring, tax_level) %>%
  nest(data = -tax_label) %>%
  mutate(clean_tax_label = map_chr(data, ~fill_ss_tbl(.x))) %>%
  unnest(clean_tax_label) %>%
  select(-data)


left_join(database_tax_label, clean_tax_labels_lookup, by = "tax_label") %>%
  select(id, clean_tax_label) %>%
  write_tsv(file="silva.full_v138_2.tax", quote = "none", col_names = FALSE)

Building the SEED references

The first thing to note is that SILVA does not release their SEED; it is private. By screening through the ARB databases we can attempt to recreate it. Our previous publications show that classify.seqs with the recreated SEED does an excellent job of realigning sequences to look like they would if you used SINA and the true SEED. Now we want to try to figure out which sequences are part of the seed. Earlier, when we exported the sequences from ARB, we included the align_ident_slv field from the database in our output. Let’s generate an accnos file that contains the names of the sequences with 100% to the SEED database and then use mothur to generate SEED fasta and taxonomy files. While we’re at it we’ll also generate the nr_238 taxonomy file as well. The following code will be run from within a bash terminal:

grep ">" silva.nr_v138_2.align | cut -f 1,2 | grep "\t100" | cut -f 1 | cut -c 2- > silva.seed_v138.accnos
mothur "#get.seqs(fasta=silva.nr_v138_2.align, taxonomy=silva.full_v138_2.tax, accnos=silva.seed_v138.accnos)"
mv silva.nr_v138_2.pick.align silva.seed_v138_2.align
mv silva.full_v138_2.pick.tax silva.seed_v138_2.tax

cp silva.full_v138_2.tax silva.nr_v138_2.tax

Taxonomic representation

Let’s look to see how many different taxa we have for each taxonomic level within the silva.nr_v138_2.tax, silva.nr_v138_2.tax. To do this we’ll run the following in R:

getNumTaxaNames <- function(file, kingdom){
  taxonomy <- read.table(file=file, row.names=1)
  sub.tax <- as.character(taxonomy[grepl(kingdom, taxonomy[,1]),])

  phyla <- as.vector(levels(as.factor(gsub("[^;]*;([^;]*;).*", "\\1", sub.tax))))
  phyla <- sum(!grepl(kingdom, phyla))

  class <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  class <- sum(!grepl(kingdom, class))

  order <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  order <- sum(!grepl(kingdom, order))

  family <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  family <- sum(!grepl(kingdom, family))

  genus <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  genus <- sum(!grepl(kingdom, genus))

  n.seqs <- length(sub.tax)
  return(c(phyla=phyla, class=class, order=order, family=family, genus=genus, n.seqs=n.seqs))
}

kingdoms <- c("Bacteria", "Archaea", "Eukaryota")
tax.levels <- c("phyla", "class", "order", "family", "genus", "n.seqs")

nr.file <- "silva.nr_v138_2.tax"
nr.matrix <- matrix(rep(0,18), nrow=3)
nr.matrix[1,] <- getNumTaxaNames(nr.file, kingdoms[1])
nr.matrix[2,] <- getNumTaxaNames(nr.file, kingdoms[2])
nr.matrix[3,] <- getNumTaxaNames(nr.file, kingdoms[3])
rownames(nr.matrix) <- kingdoms
colnames(nr.matrix) <- tax.levels
nr.matrix
#           phyla class order family genus n.seqs
# Bacteria     96   249   660   1208  4554 145520
# Archaea      14    32    62    109   251   3744
# Eukaryota   116   343  1077   1863  2694  15032

seed.file <- "silva.seed_v138_2.tax"
seed.matrix <- matrix(rep(0,18), nrow=3)
seed.matrix[1,] <- getNumTaxaNames(seed.file, kingdoms[1])
seed.matrix[2,] <- getNumTaxaNames(seed.file, kingdoms[2])
seed.matrix[3,] <- getNumTaxaNames(seed.file, kingdoms[3])
rownames(seed.matrix) <- kingdoms
colnames(seed.matrix) <- tax.levels
seed.matrix
#          phyla class order family genus n.seqs
#Bacteria     63   148   344    615  1547   6714
#Archaea       8    19    29     38    61    132
#Eukaryota    48   120   311    602   883   1850

seed.matrix / nr.matrix
#              phyla     class     order    family     genus     n.seqs
#Bacteria  0.6562500 0.5943775 0.5212121 0.5091060 0.3397014 0.04613799
#Archaea   0.5714286 0.5937500 0.4677419 0.3486239 0.2430279 0.03525641
#Eukaryota 0.4137931 0.3498542 0.2887651 0.3231347 0.3277654 0.12307078

The Archaea take a beating and recall they lost a bunch of sequences in the initial steps since many of the arachaeal sequences in SILVA are between 900 and 1200 nt long. If you are interested in analyzing the Archaea and the Eukaryota, I would suggest duplicating my efforts here but modify the screen.seqs and pcr.seqs steps to target your region of interest.

Finally, we want to compress the resulting alignment and this README file into the full length and SEED archives using commands in the bash terminal:

tar cvzf silva.nr_v138_2.tgz silva.nr_v138_2.tax silva.nr_v138_2.align README.md
tar cvzf silva.seed_v138_2.tgz silva.seed_v138_2.tax silva.seed_v138_2.align README.md

Application

So… which to use for what application? If you have the RAM, I’d suggest using silva.nr_v138_2.align in align.seqs. It took about 10 minutes to read in the database file and a minute or so to align a 1000 full-length sequences if you only use a single processor. Here is an example workflow for use within mothur that will get you the V4 region of the 16S rRNA gene:

mothur "#pcr.seqs(fasta=silva.nr_v138_2.align, start=11894, end=25319, keepdots=F);
        unique.seqs()"

This will get you down to 106,985 unique sequences to then align against. Other tricks to consider would be to use get.lineage to pull out the reference sequences that are from the Bacteria, this will probably only reduce the size of the database by ~10%. You could also try using filter.seqs with vertical=T; however, that might be problematic if there are insertions in your sequences (can’t know a priori). It’s likely that you can just use the silva.seed_v138_2.align reference for aligning. For classifying sequences, I would strongly recommend using the silva.nr_v138_2.align and silva.nr_v138_2.tax references after running pcr.seqs on silva.nr_v138_2.align. I probably wouldn’t advise using unique.seqs on the output.

Legalese

If you are going to use the files generated in this README, you should be aware that this release is available under a CC-BY license.

README for the greengenes2 2020_10 reference files

Tue, 25 Jun 2024 00:00:00 +0000

The biocore group released an updated version of the greengenes taxonomy in October 2022, which was published in Nature Biotechnology. If you use these files, you should cite McDonald et al.

Here is how I generated the mothur-compatible greengenes2 files.

To download the files we want two qza files (these are zip files that are used with QIIME 2). We can download them, extract them, and get them in an easy to find location using these bash commands:

wget --no-check-certificate https://ftp.microbio.me/greengenes_release/2022.10/2022.10.backbone.full-length.fna.qza
wget --no-check-certificate https://ftp.microbio.me/greengenes_release/2022.10/2022.10.backbone.tax.qza

unzip 2022.10.backbone.full-length.fna.qza
unzip 2022.10.backbone.tax.qza

mv */data/dna-sequences.fasta dna-sequences.fasta
mv */data/taxonomy.tsv taxonomy.tsv

Then in R we will read in the fasta and taxonomy files, make sure they’re in the correct order and polish the taxonomy strings to work with mothur:

library(tidyverse)
library(vroom)

fasta_fname <- "dna-sequences.fasta"
tax_fname <- "taxonomy.tsv"

f <- vroom_lines(fasta_fname)
indices <- seq_along(f)

seq_data <- tibble(id = f[indices %% 2 == 1],
      seq = f[indices %% 2 == 0]) %>%
  mutate(id = stringi::stri_replace_first_fixed(id, ">", ""))


tax_data <- vroom(tax_fname, delim = "\t",
                  col_names = c("id", "taxonomy"),
                  col_types = "cc",
                  skip = 1)


s_t <- anti_join(seq_data, tax_data, by = "id") %>% nrow(.) == 0
t_s <- anti_join(tax_data, seq_data, by = "id") %>% nrow(.) == 0
stopifnot(s_t, t_s)

seq_tax_data <- inner_join(seq_data, tax_data, by = "id")

Let’s look at the data a bit…

parsed_tax <- seq_tax_data %>%
  select(id, taxonomy) %>%
  separate(taxonomy, sep = "; ", into = c("k", "p", "c", "o", "f", "g", "s"))

count(parsed_tax, k)

There are 329,175 bacterial and 2,094 archaeal sequences. Here is the number of kingdoms through species included in the reference and the number of taxa at each level that only has one sequence.

parsed_tax %>%
  pivot_longer(-id) %>%
  filter(!str_detect(value, "__$")) %>%
  nest(data = -name) %>%
  mutate(summary = map_dfr(data, \(x){
    z <- count(x, value)
    tibble(n_taxa = nrow(z), n_singletons = sum(z$n == 1))
  })) %>%
  select(name, summary) %>%
  unnest(summary)

# A tibble: 7 × 3
  name  n_taxa n_singletons
  <chr>  <int>        <int>
k          2            0
p        131            2
c        342           22
o       1054          126
f       2224          376
g       8019         2006
s      22929        10278

Here’s the number of sequences with a name at each taxonomic level

parsed_tax %>%
  pivot_longer(-id) %>%
  mutate(name = factor(name, levels = c("k", "p", "c", "o", "f", "g", "s"))) %>%
  filter(!str_detect(value, "__$")) %>%
  count(name)

# A tibble: 7 × 2
  name       n
  <fct>  <int>
k     331269
p     331216
c     331006
o     328980
f     322566
g     291792
s     201702

About 61% of sequences have a species level name. Not all sequences have genera-level names and not all genera have species-level names.

parsed_tax %>%
  select(g, s) %>%
  distinct() %>%
  summarize(n_species = sum(s != "s__"), .by = g) %>% 
  count(n_species, name = "n_genera") %>%
  select(n_genera, n_species) %>%
  print(n = Inf)

   n_genera n_species
      <int>     <int>
      3         0
   5373         1
   1026         2
    487         3
    275         4
    169         5
    113         6
     92         7
     74         8
     45         9
     42        10
     44        11
     31        12
     23        13
     21        14
     17        15
     13        16
      5        17
     17        18
     11        19
      4        20
      1        21
     12        22
      7        23
      8        24
      7        25
      5        26
      6        27
      4        28
      1        29
      7        30
      4        31
      2        32
      5        33
      5        34
      3        37
      4        38
      2        39
      2        40
      1        41
      2        42
      3        43
      1        44
      1        45
      1        46
      1        47
      1        48
      1        50
      5        52
      1        56
      1        57
      2        59
      1        61
      2        66
      1        71
      1        72
      2        73
      1        74
      3        76
      1        79
      1        80
      1        83
      1        84
      1        86
      1        89
      1        92
      1       104
      1       107
      1       119
      1       120
      2       139
      1       150
      1       165
      1       199
      1       267
      1       456

This tells us that there are 3 genera with no species and 5373 with only one species; there’s also one genus with 456 species. This tells me that if you pick one of the 5373 singleton genera, you will definitely get that species - that’s about two-thirds of the genera. I worry that this will give the sense of an unrealistic sense of specificity for sequences. With that in mind, I will post the version of the greengenes2 database without species-level names. The code for generating the results with the species-level names is also included below in case you want to go out on that limb.

Let’s do the one with species-level names first:

seq_tax_data %>%
  select(id, taxonomy) %>%
  mutate(
    taxonomy = stringi::stri_replace_all_regex(taxonomy, " ", ""),
    taxonomy = stringi::stri_replace_all_regex(taxonomy, "$", ";")
  ) %>%
  write_tsv("greengenes2_2020_10.w_sp.taxonomy", col_names = FALSE)

seq_tax_data %>%
  select(id, seq) %>%
  mutate(id = stringi::stri_replace_first_regex(id, "^", ">")) %>%
  unite(fasta, id, seq, sep = "\n") %>%
  write_tsv("greengenes2_2020_10.w_sp.fasta", col_names = FALSE)

Let’s do the one without species-level names next:

seq_tax_data %>%
  select(id, taxonomy) %>%
  mutate(
    taxonomy = stringi::stri_replace_all_regex(taxonomy, " ", ""),
    taxonomy = stringi::stri_replace_all_regex(taxonomy, "s__.*", "")
  ) %>%
  write_tsv("greengenes2_2020_10.wo_sp.taxonomy", col_names = FALSE)

seq_tax_data %>%
  select(id, seq) %>%
  mutate(id = stringi::stri_replace_first_regex(id, "^", ">")) %>%
  unite(fasta, id, seq, sep = "\n") %>%
  write_tsv("greengenes2_2020_10.wo_sp.fasta", col_names = FALSE)

Let’s package it all together…

mkdir greengenes2_2020_10.wo_sp
mv greengenes2_2020_10.wo_sp.* greengenes2_2020_10.wo_sp
cp README.md greengenes2_2020_10.wo_sp
tar cvzf greengenes2_2020_10.wo_sp.tgz greengenes2_2020_10.wo_sp

README for the RDP v19 reference files

Tue, 12 Mar 2024 00:00:00 +0000

The good people at the RDP have released a new version of the RDP database. A little bit of tweaking is needed to get their files to be compatible with mothur. This README document describes the process that I used to generate the mothur-compatible reference files. The original files are available from the RDPs sourceforge server and were used as the starting point for this README.

The release notes indicate the following:

The Bacteria and Archaea hierarchy model used by RDP Classifier has been updated to training set No. 19. The new version has over 600 new genera and 2500 new species added since last version No. 18 released in July 2020. The information that is used to update the RDP taxonomy to training set version No. 19, and RDP Classifier version 2.14 came from publicly available scientific articles and public sequence repository, mostly from International Journal of Systematic and Evolutionary Microbiology (IJSEM), the All-Species Living Tree Project (LTP) and GenBank.

It is worth noting that most of the phyla have new names, according to article “ Oren A, Garrity GM. Valid publication of the names of forty-two phyla of prokaryotes. Int J Syst Evol Microbiol. 2021 Oct;71(10). doi: 10.1099/ijsem.0.005056. PMID: 34694987.”

Let’s get going…

rm -rf RDPClassifier_16S_trainsetNo19_rawtrainingdata*

wget -N http://sourceforge.net/projects/rdp-classifier/files/RDP_Classifier_TrainingData/RDPClassifier_16S_trainsetNo19_rawtrainingdata.zip
unzip -o RDPClassifier_16S_trainsetNo19_rawtrainingdata.zip
mv RDPClassifier_16S_trainsetNo19_rawtrainingdata/* ./

Now we’d like to start to form the taxonomy file and the fasta file that will be our reference. Again, using bash commands…

grep ">" trainset19_072023_speciesrank.fa | cut -c 2- > trainset19_072023_rmdup.tax
cp trainset19_072023_speciesrank.fa trainset19_072023.rdp.fasta

Next, we’d like to get our taxonomy file properly formatted. First we’ll read in the taxonomy data. Then we’ll output the taxonomy data to a file we’ll call trainset19_072023.rdp.tax to have a consistent naming scheme with previous versions of those files. The following steps are done in R…

library(tidyverse)

incertae_sedis <- function(x) {

  to_fix <- c(1:6)[str_detect(x, "domain__")][1]

  if(is.na(to_fix)) { return(x) }

  x[to_fix] <- if_else(str_detect(x[to_fix - 1], "incertae_sedis"),
                      x[to_fix - 1],
                      paste0(x[to_fix-1], "_incertae_sedis")
              )

  incertae_sedis(x)

}

parse_taxonomy <- function(x) {

  c(domain = str_replace(x, ".*domain__([^;]*);.*", "\\1"),
    phylum = str_replace(x, ".* phylum__([^;]*);.*", "\\1"),
    class = str_replace(x, ".* class__([^;]*);.*", "\\1"),
    order = str_replace(x, ".* order__([^;]*);.*", "\\1"),
    family = str_replace(x, ".* family__([^;]*);.*", "\\1"),
    genus = str_replace(x, ".* genus__([^;]*).*", "\\1")) %>%
                incertae_sedis() %>%
                as_tibble_row(.) %>%
                mutate(across(phylum:genus,
                              ~str_replace_all(.x,
                                          pattern = " ",
                                          replacement = "_")))

}

tax_data <- read_tsv(file="trainset19_072023_rmdup.tax",
                    col_names = c("accession", "species_strain", "taxonomy"),
                    col_types = cols(.default = col_character())) %>%
            select(accession, taxonomy)

tax_data %>%
  mutate(parsed = map(.data$taxonomy, parse_taxonomy)) %>%
  select(-taxonomy) %>%
  unnest(parsed) %>%
  mutate(taxonomy = paste(domain, phylum, class,
                          order, family, genus, "", sep = ";")) %>%
  select(accession, taxonomy) %>%
  write_tsv("trainset19_072023.rdp.tax", col_names=FALSE, quote="none")

The RDP training sets do not include mitochondria or sequences from eukaryotes. We find that it is helpful to have these sequences because we can get non-specific amplification at times and would like to be able to remove these lineages. Let’s go ahead and pull down the pds version of training set v.9 and copy those sequences over to our new training set. The following steps will be done in bash:

wget -N https://mothur.s3.us-east-2.amazonaws.com/wiki/trainset10_082014.pds.tgz
tar xvzf trainset10_082014.pds.tgz
mv trainset10_082014.pds/trainset10_082014* ./
rm -rf trainset10_082014.pds trainset10_082014.pds.tgz

Now let’s run a mothur command to pull out the extra sequences that are in the pds files:

mothur "#get.lineage(fasta=trainset10_082014.pds.fasta, taxonomy=trainset10_082014.pds.tax, taxon=Eukaryota-Mitochondria)"

This last command gets us the extra “pds” sequences that we can now use to paste on to the end of the normal RDP training set

cat trainset19_072023.rdp.tax trainset10_082014.pds.pick.tax > trainset19_072023.pds.tax
cat trainset19_072023.rdp.fasta trainset10_082014.pds.pick.fasta > trainset19_072023.pds.fasta

While we’ve got the old version of the training set, it might be nice to see what the differences are. It would have been nice for them to provide a README indicating what changed, but, well, no, they didn’t.

wc -l *.pds.tax

##   10773 trainset10_082014.pds.tax
##   24765 trainset19_072023.pds.tax
##   35538 total

Now we’re ready to compress the taxonomy files. First we do the RDP files…

mkdir trainset19_072023.rdp
cp README.* trainset19_072023.rdp.fasta trainset19_072023.rdp.tax trainset19_072023.rdp
tar cvzf trainset19_072023.rdp.tgz  trainset19_072023.rdp/*

##   a trainset19_072023.rdp/README.md
##   a trainset19_072023.rdp/trainset19_072023.rdp.fasta
##   a trainset19_072023.rdp/trainset19_072023.rdp.tax

… and then the pds files…

mkdir trainset19_072023.pds
cp README.* trainset19_072023.pds.fasta trainset19_072023.pds.tax trainset19_072023.pds
tar cvzf trainset19_072023.pds.tgz  trainset19_072023.pds/*

##   a trainset19_072023.pds/README.md
##   a trainset19_072023.pds/trainset19_072023.pds.fasta
##   a trainset19_072023.pds/trainset19_072023.pds.tax

README for the SILVA v138.1 reference files

Tue, 23 Feb 2021 00:00:00 +0000

The good people at SILVA have released a new version of the SILVA v138 database. My understanding is that this is a minor update to correct some taxonomic information. A little bit of tweaking is needed to get their files to be compatible with mothur. This README document describes the process that I used to generate the mothur-compatible reference files.

Curation of references

Getting the data in and out of the ARB database

wget -N https://www.arb-silva.de/fileadmin/arb_web_db/release_138_1/ARB_files/SILVA_138.1_SSURef_NR99_12_06_20_opt.arb.gz
gunzip SILVA_138.1_SSURef_NR99_05_01_20_opt.arb.gz
arb SILVA_138.1_SSURef_NR99_12_06_20_opt.arb

This will launch us into the arb environment with the ‘‘Ref NR 99’’ database opened. This database has 510,508 sequences within it that are not more than 99% similar to each other. The release notes for this database as well as the idea behind the non-redundant database are available from the silva website. Within arb do the following:

Click the search button
Set the first search field to ‘ARB_color’ and set it to 1. Click on the equal sign until it indicates not equal (this removes low quality reads and chimeras)
Click ‘Search’. This yielded 446,881 hits
Click the “Mark Listed Unmark Rest” button
Close the “Search and Query” box
Now click on File->export->export to external format
In this box the Export option should be set to marked, Filter to none, and Compression should be set to no.
In the field for Choose an output file name make sure the path has you in the correct working directory and enter silva.full_v138_1.fasta`.
Select a format: fasta_mothur.eft. This is a custom formatting file that I have created that includes the sequences accession number and it’s taxonomy across the top line. To create one for you will need to create fasta_mothur.eft in the $ARBHOME/lib/export/ folder with the following:
```
SUFFIX          fasta
BEGIN
>*(acc).*(name)\t*(align_ident_slv)\t*(tax_slv);
*(|export_sequence)
```
Save this as silva.full_v138_1.fasta
You can now quit arb.

Screening the sequences

mothur "#screen.seqs(fasta=silva.full_v138_1.fasta, start=1044, end=43116, maxambig=5);
        pcr.seqs(start=1044, end=43116, keepdots=T);
        degap.seqs();
        unique.seqs();"

#identify the unique sequences without regard to their alignment
grep ">" silva.full_v138_1.good.pcr.ng.unique.fasta | cut -f 1 | cut -c 2- > silva.full_v138_1.good.pcr.ng.unique.accnos

#get the unique sequences without regard to their alignment
mothur "#get.seqs(fasta=silva.full_v138_1.good.pcr.fasta, accnos=silva.full_v138_1.good.pcr.ng.unique.accnos)"

#generate alignment file
mv silva.full_v138_1.good.pcr.pick.fasta silva.nr_v138_1.align

#generate taxonomy file
grep '>' silva.nr_v138_1.align | cut -f1,3 | cut -f2 -d'>' > silva.nr_v138.full

Formatting the taxonomy files

Now we want to make sure the taxonomy file is properly formatted for use with mothur. First we want to grab the SILVA taxa mapping file by running the following in bash:

wget https://www.arb-silva.de/fileadmin/silva_databases/release_138_1/Exports/taxonomy/tax_slv_ssu_138.1.txt.gz
gunzip tax_slv_ssu_138.1.txt.gz

Thanks to Eric Collins at the University of Alaska Fairbanks, we have some nice R code to map all of the taxa names to the six Linnean levels (kingdom, phylum, class, order, family, and genus). We’ll run the following code from within R:

map.in <- read.table("tax_slv_ssu_138.1.txt",header=F,sep="\t",stringsAsFactors=F)
map.in <- map.in[,c(1,3)]
colnames(map.in) <- c("taxlabel","taxlevel")
<!-- map.in <- rbind(map.in, c("Bacteria;RsaHf231;", "phylum")) #wasn't in tax_slv_ssu_138.txt -->

#fix Escherichia nonsense
<!-- map.in$taxlevel[which(map.in$taxlabel=="Bacteria;Proteobacteria;Gammaproteobacteria;Enterobacteriales;Enterobacteriaceae;Escherichia;")] <- "genus" -->

taxlevels <- c("root","domain","major_clade","superkingdom","kingdom","subkingdom","infrakingdom","superphylum","phylum","subphylum","infraphylum","superclass","class","subclass","infraclass","superorder","order","suborder","superfamily","family","subfamily","genus")
taxabb <- c("ro","do","mc","pk","ki","bk","ik","pp","ph","bp","ip","pc","cl","bc","ic","po","or","bo","pf","fa","bf","ge")
tax.mat <- matrix(data="",nrow=nrow(map.in),ncol=length(taxlevels))
tax.mat[,1] <- "root"
colnames(tax.mat) <- taxlevels

outlevels <- c("domain","phylum","class","order","family","genus")

for(i in 1:nrow(map.in)) {
	taxname <- unlist(strsplit(as.character(map.in[i,1]), split=';'))
	#print(taxname);

	while ( length(taxname) > 0) {
		#regex to look for exact match

		tax.exp <- paste(paste(taxname,collapse=";"),";",sep="")
		tax.match <- match(tax.exp,map.in$taxlabel)
		tax.mat[i,map.in[tax.match,2]] <- tail(taxname,1)
		taxname <- head(taxname,-1)
	}
}

for(i in 1:nrow(tax.mat)) {
	#this fills in the empty gaps by using the closest higher taxonomic level appended with an abbreviation for the current taxonomic level
	#if you don't want this behavior, cut it out
	for(j in 1:ncol(tax.mat)) {
		if(tax.mat[i,j] < 0) { tax.mat[i,j] <- paste(tmptax,taxabb[j],sep="_")}
		else { tmptax <- tax.mat[i,j]}
	}

	#this maps the new name to the input taxonomic levels
	map.in[i,"taxout"] <- paste(paste(tax.mat[i,outlevels],collapse=";"),";",sep="")
}

# replace spaces with underscores
map.in$taxout <- gsub(" ","_",map.in$taxout)

# bring in the old taxonomic levels from SILVA and remap them using the new levels
tax.in <- read.table("silva.nr_v138.full",header=F,stringsAsFactors=F,sep="\t")
colnames(tax.in) <- c("taxid","taxlabel")

# Following line corrects the Bacteria;Bacteroidetes;Bacteroidia;Flavobacteriales;Flavobacteriaceae;Polaribacter;Polaribacter; problem
<!-- tax.in$taxlabel <- gsub("Polaribacter;Polaribacter;", "Polaribacter;", tax.in$taxlabel) -->
tax.in$taxlabel <- gsub(";[[:space:]]+$", ";", tax.in$taxlabel)

tax.in$id <- 1:nrow(tax.in)

tax.write <- merge(tax.in,map.in,all.x=T,sort=F)
tax.write <- tax.write[order(tax.write$id),]


#we want to see whether everything has 6 taxonomic level (kingdom to genus)
getDepth <- function(taxonString){
	initial <- nchar(taxonString)
	removed <- nchar(gsub(";", "", taxonString))
	return(initial-removed)
}

depth <- getDepth(tax.write$taxout)
summary(depth) #should all be 6 and there should be no NAs
bacteria <- grepl("Bacteria;", tax.write$taxout)
archaea <- grepl("Archaea;", tax.write$taxout)
eukarya <- grepl("Eukaryota;", tax.write$taxout)

tax.write[depth > 6 & bacteria,] #if zero, we're good to go
tax.write[depth > 6 & archaea,]  #if zero, we're good to go
tax.write[depth > 6 & eukarya,]  #if zero, we're good to go

write.table(tax.write[,c("taxid","taxout")], file="silva.full_v138_1.tax",sep="\t",row.names=F,quote=F,col.names=F)

Building the SEED references

The first thing to note is that SILVA does not release their SEED; it is private. By screening through the ARB databases we can attempt to recreate it. Our previous publications show that classify.seqs with the recreated SEED does an excellent job of realigning sequences to look like they would if you used SINA and the true SEED. Now we want to try to figure out which sequences are part of the seed. Earlier, when we exported the sequences from ARB, we included the align_ident_slv field from the database in our output. Let’s generate an accnos file that contains the names of the sequences with 100% to the SEED database and then use mothur to generate SEED fasta and taxonomy files. While we’re at it we’ll also generate the nr_138 taxonomy file as well. The following code will be run from within a bash terminal:

grep ">" silva.nr_v138_1.align | cut -f 1,2 | grep "\t100" | cut -f 1 | cut -c 2- > silva.seed_v138.accnos
mothur "#get.seqs(fasta=silva.nr_v138_1.align, taxonomy=silva.full_v138_1.tax, accnos=silva.seed_v138.accnos)"
mv silva.nr_v138.pick.align silva.seed_v138_1.align
mv silva.full_v138_1.pick.tax silva.seed_v138_1.tax

mothur "#get.seqs(taxonomy=silva.full_v138_1.tax, accnos=silva.full_v138_1.good.pcr.ng.unique.accnos)"
mv silva.full_v138_1.pick.tax silva.nr_v138_1.tax

Taxonomic representation

Let’s look to see how many different taxa we have for each taxonomic level within the silva.nr_v138_1.tax, silva.nr_v138_1.tax. To do this we’ll run the following in R:

getNumTaxaNames <- function(file, kingdom){
  taxonomy <- read.table(file=file, row.names=1)
  sub.tax <- as.character(taxonomy[grepl(kingdom, taxonomy[,1]),])

  phyla <- as.vector(levels(as.factor(gsub("[^;]*;([^;]*;).*", "\\1", sub.tax))))
  phyla <- sum(!grepl(kingdom, phyla))

  class <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  class <- sum(!grepl(kingdom, class))

  order <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  order <- sum(!grepl(kingdom, order))

  family <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  family <- sum(!grepl(kingdom, family))

  genus <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  genus <- sum(!grepl(kingdom, genus))

  n.seqs <- length(sub.tax)
  return(c(phyla=phyla, class=class, order=order, family=family, genus=genus, n.seqs=n.seqs))
}

kingdoms <- c("Bacteria", "Archaea", "Eukaryota")
tax.levels <- c("phyla", "class", "order", "family", "genus", "n.seqs")

nr.file <- "silva.nr_v138_1.tax"
nr.matrix <- matrix(rep(0,18), nrow=3)
nr.matrix[1,] <- getNumTaxaNames(nr.file, kingdoms[1])
nr.matrix[2,] <- getNumTaxaNames(nr.file, kingdoms[2])
nr.matrix[3,] <- getNumTaxaNames(nr.file, kingdoms[3])
rownames(nr.matrix) <- kingdoms
colnames(nr.matrix) <- tax.levels
nr.matrix
#          phyla class order family genus n.seqs
#Bacteria     87   238   631   1139  3955 128884
#Archaea      15    33    57     97   222   2846
#Eukaryota    92   243   644    871  2682  14871


seed.file <- "silva.seed_v138_1.tax"
seed.matrix <- matrix(rep(0,18), nrow=3)
seed.matrix[1,] <- getNumTaxaNames(seed.file, kingdoms[1])
seed.matrix[2,] <- getNumTaxaNames(seed.file, kingdoms[2])
seed.matrix[3,] <- getNumTaxaNames(seed.file, kingdoms[3])
rownames(seed.matrix) <- kingdoms
colnames(seed.matrix) <- tax.levels
seed.matrix
#          phyla class order family genus n.seqs
#Bacteria     51   123   299    523  1182   5736
#Archaea       7    17    23     30    44     81
#Eukaryota    40    98   272    422   855   1824

seed.matrix / nr.matrix
#              phyla     class     order    family     genus     n.seqs
#Bacteria  0.5862069 0.5168067 0.4738510 0.4591747 0.2988622 0.04450514
#Archaea   0.4666667 0.5151515 0.4035088 0.3092784 0.1981982 0.02846100
#Eukaryota 0.4347826 0.4032922 0.4223602 0.4845006 0.3187919 0.12265483

Finally, we want to compress the resulting alignment and this README file into the full length and SEED archives using commands in the bash terminal:

tar cvzf silva.nr_v138_1.tgz silva.nr_v138_1.tax silva.nr_v138_1.align README.md
tar cvzf silva.seed_v138_1.tgz silva.seed_v138_1.tax silva.seed_v138_1.align README.md

Application

So… which to use for what application? If you have the RAM, I’d suggest using silva.nr_v138_1.align in align.seqs. It took about 10 minutes to read in the database file and a minute or so to align a 1000 full-length sequences if you only use a single processor. Here is an example workflow for use within mothur that will get you the V4 region of the 16S rRNA gene:

mothur "#pcr.seqs(fasta=silva.nr_v138_1.align, start=11894, end=25319, keepdots=F);
        unique.seqs()"

This will get you down to 106,985 unique sequences to then align against. Other tricks to consider would be to use get.lineage to pull out the reference sequences that are from the Bacteria, this will probably only reduce the size of the database by ~10%. You could also try using filter.seqs with vertical=T; however, that might be problematic if there are insertions in your sequences (can’t know a priori). It’s likely that you can just use the silva.seed_v138_1.align reference for aligning. For classifying sequences, I would strongly recommend using the silva.nr_v138_1.align and silva.nr_v138_1.tax references after running pcr.seqs on silva.nr_v138_1.align. I probably wouldn’t advise using unique.seqs on the output.

Legalese

If you are going to use the files generated in this README, you should be aware that this release is available under a CC-BY license.

README for the RDP v18 reference files

Thu, 04 Feb 2021 00:00:00 +0000

The release notes indicate the following:

The Bacteria and Archaea hierarchy model used by RDP Classifier has been updated to training set No. 18. The new version has over 800 new genera and 4000 new species added. Major rearrangements for Classifier training set No. 18 include the following: (go check out the release notes that are linked above for the list of changes).

Let’s get going…

rm -rf RDPClassifier_16S_trainsetNo18_rawtrainingdata*

wget -N http://sourceforge.net/projects/rdp-classifier/files/RDP_Classifier_TrainingData/RDPClassifier_16S_trainsetNo18_rawtrainingdata.zip
unzip -o RDPClassifier_16S_trainsetNo18_rawtrainingdata.zip
mv RDPClassifier_16S_trainsetNo18_rawtrainingdata/* ./

Now we’d like to start to form the taxonomy file and the fasta file that will be our reference. Again, using bash commands…

mv trainset18_062020.fa trainset18_062020.rdp.fasta
grep ">" trainset18_062020.rdp.fasta | cut -c 2- > trainset18_062020_rmdup.tax

Next, we’d like to get our taxonomy file properly formatted. First we’ll read in the taxonomy data. The following steps are done in R…

tax_file <- scan(file="trainset18_062020_rmdup.tax", what="", sep="\n", quiet=TRUE)

accession <- gsub("^(\\S*).*", "\\1", tax_file) #some are separated by tabs or spaces or both

taxonomy <- gsub(".*(Root.*)", "\\1", tax_file)
taxonomy <- gsub(" ", "_", taxonomy)	#remove spaces and replace with '_'
taxonomy <- gsub("\t", "", taxonomy)	#remove extra tab characters
taxonomy <- gsub("[^;]*_incertae_sedis$", "", taxonomy)
taxonomy <- gsub('\"', '', taxonomy) #remove quote marks

The RDP inserts a variety of sub taxonomic levels (e.g. suborder) that will get in the way of us having a consistent number of taxonomic levels for our analyses. Let’s use the data in trainset18_db_taxid.txt to remove these extra taxonomic levels:

levels <- read.table(file="trainset18_db_taxid.txt", sep="*", stringsAsFactors=FALSE)
subs <- levels[grep("sub", levels$V5),]
sub.names <- subs$V2

tax.split <- strsplit(taxonomy, split=";")

remove.subs <- function(tax.vector){
	return(tax.vector[which(!tax.vector %in% sub.names)])
}

no.subs <- lapply(tax.split, remove.subs)
no.subs.str <- unlist(lapply(no.subs, paste, collapse=";"))
no.subs.str <- gsub("^Root;(.*)$", "\\1;", no.subs.str)

Finally, we can output the taxonomy data to a file we’ll call trainset18_062020.rdp.tax to have a consistent naming scheme with previous versions of those files:

write.table(cbind(as.character(accession), no.subs.str), "trainset18_062020.rdp.tax", row.names=F, col.names=F, quote=F, sep="\t")

wget -N https://mothur.s3.us-east-2.amazonaws.com/wiki/trainset10_082014.pds.tgz
tar xvzf trainset10_082014.pds.tgz
mv trainset10_082014.pds/trainset10_082014* ./
rm -rf trainset10_082014.pds trainset10_082014.pds.tgz

Now let’s run a mothur command to pull out the extra sequences that are in the pds files:

mothur "#get.lineage(fasta=trainset10_082014.pds.fasta, taxonomy=trainset10_082014.pds.tax, taxon=Eukaryota-Mitochondria)"

This last command gets us the extra “pds” sequences that we can now use to paste on to the end of the normal RDP training set

cat trainset18_062020.rdp.tax trainset10_082014.pds.pick.tax > trainset18_062020.pds.tax
cat trainset18_062020.rdp.fasta trainset10_082014.pds.pick.fasta > trainset18_062020.pds.fasta

wc -l *.pds.tax

## 10773 trainset10_082014.pds.tax
## 21318 trainset18_062020.pds.tax
## 32091 total

Now we’re ready to compress the taxonomy files. First we do the RDP files…

mkdir trainset18_062020.rdp
cp README.* trainset18_062020.rdp.fasta trainset18_062020.rdp.tax trainset18_062020.rdp
tar cvzf trainset18_062020.rdp.tgz  trainset18_062020.rdp/*

## a trainset18_062020.rdp/README.md
## a trainset18_062020.rdp/trainset18_062020.rdp.fasta
## a trainset18_062020.rdp/trainset18_062020.rdp.tax

… and then the pds files…

mkdir trainset18_062020.pds
cp README.* trainset18_062020.pds.fasta trainset18_062020.pds.tax trainset18_062020.pds
tar cvzf trainset18_062020.pds.tgz  trainset18_062020.pds/*

## a trainset18_062020.pds/README.md
## a trainset18_062020.pds/trainset18_062020.pds.fasta
## a trainset18_062020.pds/trainset18_062020.pds.tax

README for the SILVA v138 reference files

Wed, 04 Mar 2020 00:00:00 +0000

The good people at SILVA have released a new version of the SILVA database. A little bit of tweaking is needed to get their files to be compatible with mothur. This README document describes the process that I used to generate the mothur-compatible reference files.

Getting the data in and out of the ARB database

wget -N https://www.arb-silva.de/fileadmin/silva_databases/release_138/ARB_files/SILVA_138_SSURef_NR99_05_01_20_opt.arb.gz
gunzip SILVA_138_SSURef_NR99_05_01_20_opt.arb.gz
arb SILVA_138_SSURef_NR99_05_01_20_opt.arb

This will launch us into the arb environment with the ‘‘Ref NR 99’’ database opened. This database has 510,984 sequences within it that are not more than 99% similar to each other. The release notes for this database as well as the idea behind the non-redundant database are available from the silva website. Within arb do the following:

Click the search button
Set the first search field to ‘ARB_color’ and set it to 1. Click on the equal sign until it indicates not equal (this removes low quality reads and chimeras)
Click ‘Search’. This yielded 447,349 hits
Click the “Mark Listed Unmark Rest” button
Close the “Search and Query” box
Now click on File->export->export to external format
In this box the Export option should be set to marked, Filter to none, and Compression should be set to no.
In the field for Choose an output file name make sure the path has you in the correct working directory and enter silva.full_v138.fasta.
Select a format: fasta_mothur.eft. This is a custom formatting file that I have created that includes the sequences accession number and it’s taxonomy across the top line. To create one for you will need to create fasta_mothur.eft in the $ARBHOME/lib/export/ folder with the following:
```
SUFFIX          fasta    
BEGIN    
>*(acc).*(name)\t*(align_ident_slv)\t*(tax_slv);    
*(|export_sequence)    
```
Save this as silva.full_v138.fasta
You can now quit arb.

Screening the sequences

mothur "#screen.seqs(fasta=silva.full_v138.fasta, start=1044, end=43116, maxambig=5, processors=8);
        pcr.seqs(start=1044, end=43116, keepdots=T);
        degap.seqs();
        unique.seqs();"

#identify the unique sequences without regard to their alignment
grep ">" silva.full_v138.good.pcr.ng.unique.fasta | cut -f 1 | cut -c 2- > silva.full_v138.good.pcr.ng.unique.accnos

#get the unique sequences without regard to their alignment
mothur "#get.seqs(fasta=silva.full_v138.good.pcr.fasta, accnos=silva.full_v138.good.pcr.ng.unique.accnos)"

#generate alignment file
mv silva.full_v138.good.pcr.pick.fasta silva.nr_v138.align

#generate taxonomy file
grep '>' silva.nr_v138.align | cut -f1,3 | cut -f2 -d'>' > silva.nr_v138.full

Formatting the taxonomy files

Now we want to make sure the taxonomy file is properly formatted for use with mothur. First we want to grab the SILVA taxa mapping file by running the following in bash:

wget https://www.arb-silva.de/fileadmin/silva_databases/release_138/Exports/taxonomy/tax_slv_ssu_138.txt.gz
gunzip tax_slv_ssu_138.txt.gz

map.in <- read.table("tax_slv_ssu_138.txt",header=F,sep="\t",stringsAsFactors=F)
map.in <- map.in[,c(1,3)]
colnames(map.in) <- c("taxlabel","taxlevel")
map.in <- rbind(map.in, c("Bacteria;RsaHf231;", "phylum")) #wasn't in tax_slv_ssu_138.txt

#fix Escherichia nonsense
map.in$taxlevel[which(map.in$taxlabel=="Bacteria;Proteobacteria;Gammaproteobacteria;Enterobacteriales;Enterobacteriaceae;Escherichia;")] <- "genus"

taxlevels <- c("root","domain","major_clade","superkingdom","kingdom","subkingdom","infrakingdom","superphylum","phylum","subphylum","infraphylum","superclass","class","subclass","infraclass","superorder","order","suborder","superfamily","family","subfamily","genus")
taxabb <- c("ro","do","mc","pk","ki","bk","ik","pp","ph","bp","ip","pc","cl","bc","ic","po","or","bo","pf","fa","bf","ge")
tax.mat <- matrix(data="",nrow=nrow(map.in),ncol=length(taxlevels))
tax.mat[,1] <- "root"
colnames(tax.mat) <- taxlevels

outlevels <- c("domain","phylum","class","order","family","genus")

for(i in 1:nrow(map.in)) {
	taxname <- unlist(strsplit(as.character(map.in[i,1]), split=';'))
	#print(taxname);

	while ( length(taxname) > 0) {
		#regex to look for exact match

		tax.exp <- paste(paste(taxname,collapse=";"),";",sep="")
		tax.match <- match(tax.exp,map.in$taxlabel)
		tax.mat[i,map.in[tax.match,2]] <- tail(taxname,1)
		taxname <- head(taxname,-1)
	}
}

for(i in 1:nrow(tax.mat)) {
	#this fills in the empty gaps by using the closest higher taxonomic level appended with an abbreviation for the current taxonomic level
	#if you don't want this behavior, cut it out
	for(j in 1:ncol(tax.mat)) {
		if(tax.mat[i,j] < 0) { tax.mat[i,j] <- paste(tmptax,taxabb[j],sep="_")}
		else { tmptax <- tax.mat[i,j]}
	}

	#this maps the new name to the input taxonomic levels
	map.in[i,"taxout"] <- paste(paste(tax.mat[i,outlevels],collapse=";"),";",sep="")
}

# replace spaces with underscores
map.in$taxout <- gsub(" ","_",map.in$taxout)

# bring in the old taxonomic levels from SILVA and remap them using the new levels
tax.in <- read.table("silva.nr_v138.full",header=F,stringsAsFactors=F,sep="\t")
colnames(tax.in) <- c("taxid","taxlabel")

# Following line corrects the Bacteria;Bacteroidetes;Bacteroidia;Flavobacteriales;Flavobacteriaceae;Polaribacter;Polaribacter; problem
tax.in$taxlabel <- gsub("Polaribacter;Polaribacter;", "Polaribacter;", tax.in$taxlabel)
tax.in$taxlabel <- gsub(";[[:space:]]+$", ";", tax.in$taxlabel)

tax.in$id <- 1:nrow(tax.in)

tax.write <- merge(tax.in,map.in,all.x=T,sort=F)
tax.write <- tax.write[order(tax.write$id),]


#we want to see whether everything has 6 taxonomic level (kingdom to genus)
getDepth <- function(taxonString){
	initial <- nchar(taxonString)
	removed <- nchar(gsub(";", "", taxonString))
	return(initial-removed)
}

depth <- getDepth(tax.write$taxout)
summary(depth) #should all be 6 and there should be no NAs
bacteria <- grepl("Bacteria;", tax.write$taxout)
archaea <- grepl("Archaea;", tax.write$taxout)
eukarya <- grepl("Eukaryota;", tax.write$taxout)

tax.write[depth > 6 & bacteria,] #if zero, we're good to go
tax.write[depth > 6 & archaea,]  #if zero, we're good to go
tax.write[depth > 6 & eukarya,]  #if zero, we're good to go

write.table(tax.write[,c("taxid","taxout")],file="silva.full_v138.tax",sep="\t",row.names=F,quote=F,col.names=F)

Building the SEED references

The first thing to note is that SILVA does not release their SEED; it is private. By screening through the ARB databases we can attempt to recreate it. Our previous publications show that classify.seqs with the recreated SEED does an excellent job of realigning sequences to look like they would if you used SINA and the true SEED. Now we want to try to figure out which sequences are part of the seed. Earlier, when we exported the sequences from ARB, we included the align_ident_slv field from the database in our output. Let’s generate an accnos file that contains the names of the sequences with 100% to the SEED database and then use mothur to generate SEED fasta and taxonomy files. While we’re at it we’ll also generate the nr_138 taxonomy file as well. The following code will be run from within a bash terminal:

grep ">" silva.nr_v138.align | cut -f 1,2 | grep "\t100" | cut -f 1 | cut -c 2- > silva.seed_v138.accnos
mothur "#get.seqs(fasta=silva.nr_v138.align, taxonomy=silva.full_v138.tax, accnos=silva.seed_v138.accnos)"
mv silva.nr_v138.pick.align silva.seed_v138.align
mv silva.full_v138.pick.tax silva.seed_v138.tax

mothur "#get.seqs(taxonomy=silva.full_v138.tax, accnos=silva.full_v138.good.pcr.ng.unique.accnos)"
mv silva.full_v138.pick.tax silva.nr_v138.tax

Taxonomic representation

Let’s look to see how many different taxa we have for each taxonomic level within the silva.nr_v138.tax, silva.seed_v138.tax. To do this we’ll run the following in R:

getNumTaxaNames <- function(file, kingdom){
  taxonomy <- read.table(file=file, row.names=1)
  sub.tax <- as.character(taxonomy[grepl(kingdom, taxonomy[,1]),])

  phyla <- as.vector(levels(as.factor(gsub("[^;]*;([^;]*;).*", "\\1", sub.tax))))
  phyla <- sum(!grepl(kingdom, phyla))

  class <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  class <- sum(!grepl(kingdom, class))

  order <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  order <- sum(!grepl(kingdom, order))

  family <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  family <- sum(!grepl(kingdom, family))

  genus <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  genus <- sum(!grepl(kingdom, genus))

  n.seqs <- length(sub.tax)
  return(c(phyla=phyla, class=class, order=order, family=family, genus=genus, n.seqs=n.seqs))
}

kingdoms <- c("Bacteria", "Archaea", "Eukaryota")
tax.levels <- c("phyla", "class", "order", "family", "genus", "n.seqs")

nr.file <- "silva.nr_v138.tax"
nr.matrix <- matrix(rep(0,18), nrow=3)
nr.matrix[1,] <- getNumTaxaNames(nr.file, kingdoms[1])
nr.matrix[2,] <- getNumTaxaNames(nr.file, kingdoms[2])
nr.matrix[3,] <- getNumTaxaNames(nr.file, kingdoms[3])
rownames(nr.matrix) <- kingdoms
colnames(nr.matrix) <- tax.levels
nr.matrix
#          phyla class order family genus n.seqs
#Bacteria     87   239   646   1138  3897 129063
#Archaea      15    33    57     97   219   2846
#Eukaryota    91   242   557    766  1727  14887

seed.file <- "silva.seed_v138.tax"
seed.matrix <- matrix(rep(0,18), nrow=3)
seed.matrix[1,] <- getNumTaxaNames(seed.file, kingdoms[1])
seed.matrix[2,] <- getNumTaxaNames(seed.file, kingdoms[2])
seed.matrix[3,] <- getNumTaxaNames(seed.file, kingdoms[3])
rownames(seed.matrix) <- kingdoms
colnames(seed.matrix) <- tax.levels
seed.matrix
#          phyla class order family genus n.seqs
#Bacteria     51   124   312    522  1172   5741
#Archaea       7    17    23     30    44     81
#Eukaryota    40    98   238    380   727   1834

seed.matrix / nr.matrix
#              phyla     class     order    family     genus     n.seqs
#Bacteria  0.5862069 0.5188285 0.4829721 0.4586995 0.3007442 0.04448215
#Archaea   0.4666667 0.5151515 0.4035088 0.3092784 0.2009132 0.02846100
#Eukaryota 0.4395604 0.4049587 0.4272890 0.4960836 0.4209612 0.12319473

Finally, we want to compress the resulting alignment and this README file into the full length and SEED archives using commands in the bash terminal:

tar cvzf silva.nr_v138.tgz silva.nr_v138.tax silva.nr_v138.align README.md
tar cvzf silva.seed_v138.tgz silva.seed_v138.tax silva.seed_v138.align README.md

Application

So… which to use for what application? If you have the RAM, I’d suggest using silva.nr_v138.align in align.seqs. It took about 10 minutes to read in the database file and a minute or so to align a 1000 full-length sequences if you only use a single processor. Here is an example workflow for use within mothur that will get you the V4 region of the 16S rRNA gene:

mothur "#pcr.seqs(fasta=silva.nr_v138.align, start=11894, end=25319, keepdots=F, processors=8);
        unique.seqs()"

This will get you down to 107,001 unique sequences to then align against. Other tricks to consider would be to use get.lineage to pull out the reference sequences that are from the Bacteria, this will probably only reduce the size of the database by ~10%. You could also try using filter.seqs with vertical=T; however, that might be problematic if there are insertions in your sequences (can’t know a priori). It’s likely that you can just use the silva.seed_v138.align reference for aligning. For classifying sequences, I would strongly recommend using the silva.nr_v138.align and silva.nr_v138.tax references after running pcr.seqs on silva.nr_v138.align. I probably wouldn’t advise using unique.seqs on the output.

Legalese

If you are going to use the files generated in this README, you should be aware that this release is available under a CC-BY license.

README for the SILVA v132 reference files

Wed, 10 Jan 2018 00:00:00 +0000

Curation of references

Getting the data in and out of the ARB database

This README file explains how we generated the silva reference files for use with mothur’s classify.seqs and align.seqs commands. I’ll assume that you have a functioning copy of arb installed on your computer. For this README we are using version 6.0. First we need to download the database and decompress it. From the command line we do the following:

wget -N https://www.arb-silva.de/fileadmin/arb_web_db/release_132/ARB_files/SILVA_132_SSURef_NR99_13_12_17_opt.arb.gz
gunzip SILVA_132_SSURef_NR99_13_12_17_opt.arb.gz
arb SILVA_132_SSURef_NR99_13_12_17_opt.arb

This will launch us into the arb environment with the ‘‘Ref NR 99’’ database opened. This database has 695,171 sequences within it that are not more than 99% similar to each other. The release notes for this database as well as the idea behind the non-redundant database are available from the silva website. Within arb do the following:

Click the search button
Set the first search field to ‘ARB_color’ and set it to 1. Click on the equal sign until it indicates not equal (this removes low quality reads and chimeras)
Click ‘Search’. This yielded 629,211 hits
Click the “Mark Listed Unmark Rest” button
Close the “Search and Query” box
Now click on File->export->export to external format
In this box the Export option should be set to marked, Filter to none, and Compression should be set to no.
In the field for Choose an output file name make sure the path has you in the correct working directory and enter silva.full_v132.fasta`.
Select a format: fasta_mothur.eft. This is a custom formatting file that I have created that includes the sequences accession number and it’s taxonomy across the top line. To create one for you will need to create fasta_mothur.eft in the /opt/local/share/arb/lib/export/ folder with the following:
```
SUFFIX          fasta    
BEGIN    
>*(acc).*(name)\t*(align_ident_slv)\t*(tax_slv);    
*(|export_sequence)    
```
Save this as silva.full_v132.fasta
You can now quit arb.

Screening the sequences

mothur "#screen.seqs(fasta=silva.full_v132.fasta, start=1044, end=43116, maxambig=5, processors=8);
        pcr.seqs(start=1044, end=43116, keepdots=T);
        degap.seqs();
        unique.seqs();"

#identify the unique sequences without regard to their alignment
grep ">" silva.full_v132.good.pcr.ng.unique.fasta | cut -f 1 | cut -c 2- > silva.full_v132.good.pcr.ng.unique.accnos

#get the unique sequences without regard to their alignment
mothur "#get.seqs(fasta=silva.full_v132.good.pcr.fasta, accnos=silva.full_v132.good.pcr.ng.unique.accnos)"

#generate alignment file
mv silva.full_v132.good.pcr.pick.fasta silva.nr_v132.align

#generate taxonomy file
grep '>' silva.nr_v132.align | cut -f1,3 | cut -f2 -d'>' > silva.nr_v132.full

The mothur commands above do several things. First the screen.seqs command removes sequences that are not full length and have more than 5 ambiguous base calls. Note: this will remove a number of Archaea since the ARB RN reference database lets in shorter (>900 bp) archaeal 16S rRNA gene sequences. Second, pcr.seqs convert any base calls that occur before position 1044 and after 43116 to . to make them only span the region between the 27f and 1492r priming sites. Finally, it is possible that weird things happen in the alignments and so we unalign the sequences (degap.seqs) and identify the unique sequences (unique.seqs). We then convert the resulting fasta file into an accnos file so that we can go back into mothur and pull out the unique sequences from the aligned file (get.seqs).

Formatting the taxonomy files

Now we want to make sure the taxonomy file is properly formatted for use with mothur. First we want to grab the SILVA taxa mapping file by running the following in bash:

wget https://www.arb-silva.de/fileadmin/silva_databases/current/Exports/taxonomy/tax_slv_ssu_132.txt

map.in <- read.table("tax_slv_ssu_132.txt",header=F,sep="\t",stringsAsFactors=F)
map.in <- map.in[,c(1,3)]
colnames(map.in) <- c("taxlabel","taxlevel")
map.in <- rbind(map.in, c("Bacteria;RsaHf231;", "phylum")) #wasn't in tax_slv_ssu_132.txt

#fix Escherichia nonsense
map.in$taxlevel[which(map.in$taxlabel=="Bacteria;Proteobacteria;Gammaproteobacteria;Enterobacteriales;Enterobacteriaceae;Escherichia;")] <- "genus"

taxlevels <- c("root","domain","major_clade","superkingdom","kingdom","subkingdom","infrakingdom","superphylum","phylum","subphylum","infraphylum","superclass","class","subclass","infraclass","superorder","order","suborder","superfamily","family","subfamily","genus")
taxabb <- c("ro","do","mc","pk","ki","bk","ik","pp","ph","bp","ip","pc","cl","bc","ic","po","or","bo","pf","fa","bf","ge")
tax.mat <- matrix(data="",nrow=nrow(map.in),ncol=length(taxlevels))
tax.mat[,1] <- "root"
colnames(tax.mat) <- taxlevels

outlevels <- c("domain","phylum","class","order","family","genus")

for(i in 1:nrow(map.in)) {
	taxname <- unlist(strsplit(as.character(map.in[i,1]), split=';'))
	#print(taxname);

	while ( length(taxname) > 0) {
		#regex to look for exact match

		tax.exp <- paste(paste(taxname,collapse=";"),";",sep="")
		tax.match <- match(tax.exp,map.in$taxlabel)
		tax.mat[i,map.in[tax.match,2]] <- tail(taxname,1)
		taxname <- head(taxname,-1)
	}
}

for(i in 1:nrow(tax.mat)) {
	#this fills in the empty gaps by using the closest higher taxonomic level appended with an abbreviation for the current taxonomic level
	#if you don't want this behavior, cut it out
	for(j in 1:ncol(tax.mat)) {
		if(tax.mat[i,j] < 0) { tax.mat[i,j] <- paste(tmptax,taxabb[j],sep="_")}
		else { tmptax <- tax.mat[i,j]}
	}

	#this maps the new name to the input taxonomic levels
	map.in[i,"taxout"] <- paste(paste(tax.mat[i,outlevels],collapse=";"),";",sep="")
}

# replace spaces with underscores
map.in$taxout <- gsub(" ","_",map.in$taxout)

# bring in the old taxonomic levels from SILVA and remap them using the new levels
tax.in <- read.table("silva.nr_v132.full",header=F,stringsAsFactors=F,sep="\t")
colnames(tax.in) <- c("taxid","taxlabel")

# Following line corrects the Bacteria;Bacteroidetes;Bacteroidia;Flavobacteriales;Flavobacteriaceae;Polaribacter;Polaribacter; problem
tax.in$taxlabel <- gsub("Polaribacter;Polaribacter;", "Polaribacter;", tax.in$taxlabel)
tax.in$taxlabel <- gsub(";[[:space:]]+$", ";", tax.in$taxlabel)

tax.in$id <- 1:nrow(tax.in)

tax.write <- merge(tax.in,map.in,all.x=T,sort=F)
tax.write <- tax.write[order(tax.write$id),]


#we want to see whether everything has 6 taxonomic level (kingdom to genus)
getDepth <- function(taxonString){
	initial <- nchar(taxonString)
	removed <- nchar(gsub(";", "", taxonString))
	return(initial-removed)
}

depth <- getDepth(tax.write$taxout)
summary(depth) #should all be 6 and there should be no NAs
bacteria <- grepl("Bacteria;", tax.write$taxout)
archaea <- grepl("Archaea;", tax.write$taxout)
eukarya <- grepl("Eukaryota;", tax.write$taxout)

tax.write[depth > 6 & bacteria,] #good to go
tax.write[depth > 6 & archaea,]  #good to go
tax.write[depth > 6 & eukarya,]  #good to go

write.table(tax.write[,c("taxid","taxout")],file="silva.full_v132.tax",sep="\t",row.names=F,quote=F,col.names=F)

Building the SEED references

The first thing to note is that SILVA does not release their SEED; it is private. By screening through the ARB databases we can attempt to recreate it. Our previous publications show that classify.seqs with the recreated SEED does an excellent job of realigning sequences to look like they would if you used SINA and the true SEED. Now we want to try to figure out which sequences are part of the seed. Earlier, when we exported the sequences from ARB, we included the align_ident_slv field from the database in our output. Let’s generate an accnos file that contains the names of the sequences with 100% to the SEED database and then use mothur to generate SEED fasta and taxonomy files. While we’re at it we’ll also generate the nr_132 taxonomy file as well. The following code will be run from within a bash terminal:

grep ">" silva.nr_v132.align | cut -f 1,2 | grep "\t100" | cut -f 1 | cut -c 2- > silva.seed_v132.accnos
mothur "#get.seqs(fasta=silva.nr_v132.align, taxonomy=silva.full_v132.tax, accnos=silva.seed_v132.accnos)"
mv silva.nr_v132.pick.align silva.seed_v132.align
mv silva.full_v132.pick.tax silva.seed_v132.tax

mothur "#get.seqs(taxonomy=silva.full_v132.tax, accnos=silva.full_v132.good.pcr.ng.unique.accnos)"
mv silva.full_v132.pick.tax silva.nr_v132.tax

Taxonomic representation

Let’s look to see how many different taxa we have for each taxonomic level within the silva.nr_v132.tax, silva.seed_v132.tax. To do this we’ll run the following in R:

getNumTaxaNames <- function(file, kingdom){
  taxonomy <- read.table(file=file, row.names=1)
  sub.tax <- as.character(taxonomy[grepl(kingdom, taxonomy[,1]),])

  phyla <- as.vector(levels(as.factor(gsub("[^;]*;([^;]*;).*", "\\1", sub.tax))))
  phyla <- sum(!grepl(kingdom, phyla))

  class <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  class <- sum(!grepl(kingdom, class))

  order <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  order <- sum(!grepl(kingdom, order))

  family <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  family <- sum(!grepl(kingdom, family))

  genus <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  genus <- sum(!grepl(kingdom, genus))

  n.seqs <- length(sub.tax)
  return(c(phyla=phyla, class=class, order=order, family=family, genus=genus, n.seqs=n.seqs))
}

kingdoms <- c("Bacteria", "Archaea", "Eukaryota")
tax.levels <- c("phyla", "class", "order", "family", "genus", "n.seqs")

nr.file <- "silva.nr_v132.tax"
nr.matrix <- matrix(rep(0,18), nrow=3)
nr.matrix[1,] <- getNumTaxaNames(nr.file, kingdoms[1])
nr.matrix[2,] <- getNumTaxaNames(nr.file, kingdoms[2])
nr.matrix[3,] <- getNumTaxaNames(nr.file, kingdoms[3])
rownames(nr.matrix) <- kingdoms
colnames(nr.matrix) <- tax.levels
nr.matrix
#          phyla class order family genus n.seqs
#Bacteria     80   204   580   1052  3971 188247
#Archaea      11    30    52     85   210   4626
#Eukaryota    93   240   648    923  3018  20246

seed.file <- "silva.seed_v132.tax"
seed.matrix <- matrix(rep(0,18), nrow=3)
seed.matrix[1,] <- getNumTaxaNames(seed.file, kingdoms[1])
seed.matrix[2,] <- getNumTaxaNames(seed.file, kingdoms[2])
seed.matrix[3,] <- getNumTaxaNames(seed.file, kingdoms[3])
rownames(seed.matrix) <- kingdoms
colnames(seed.matrix) <- tax.levels
seed.matrix
#          phyla class order family genus n.seqs
#Bacteria     50   110   301    530  1436   8517
#Archaea       7    15    26     39    62    147
#Eukaryota    41   100   287    478  1040   2516

seed.matrix / nr.matrix
#              phyla     class     order    family     genus     n.seqs
#Bacteria  0.6250000 0.5392157 0.5189655 0.5038023 0.3616218 0.04524375
#Archaea   0.6363636 0.5000000 0.5000000 0.4588235 0.2952381 0.03177691
#Eukaryota 0.4408602 0.4166667 0.4429012 0.5178765 0.3445991 0.12427146

Finally, we want to compress the resulting alignment and this README file into the full length and SEED archives using commands in the bash terminal:

tar cvzf silva.nr_v132.tgz silva.nr_v132.tax silva.nr_v132.align README.md
tar cvzf silva.seed_v132.tgz silva.seed_v132.tax silva.seed_v132.align README.md

Application

So… which to use for what application? If you have the RAM, I’d suggest using silva.nr_v132.align in align.seqs. It took about 10 minutes to read in the database file and a minute or so to align a 1000 full-length sequences. Here is an example workflow for use within mothur that will get you the V4 region of the 16S rRNA gene:

mothur "#pcr.seqs(fasta=silva.nr_v132.align, start=11894, end=25319, keepdots=F, processors=8);
        unique.seqs()"

This will get you 139,321 unique sequences to then align against (meh.). Other tricks to consider would be to use get.lineage to pull out the reference sequences that are from the Bacteria, this will probably only reduce the size of the database by ~10%. You could also try using filter.seqs with vertical=T; however, that might be problematic if there are insertions in your sequences (can’t know a priori). It’s likely that you can just use the silva.seed_v132.align reference for aligning. For classifying sequences, I would strongly recommend using the silva.nr_v132.align and silva.nr_v132.tax references after running pcr.seqs on silva.nr_v132.align. I probably wouldn’t advise using unique.seqs on the output.

Legalese

If you are going to use the files generated in this README, you should be aware of SILVA’s dual use license. We’ll leave it to you to work out the details.

README for the SILVA v128 reference files

Wed, 22 Mar 2017 00:00:00 +0000

Curation of references

Getting the data in and out of the ARB database

wget -N https://www.arb-silva.de/fileadmin/arb_web_db/release_128/ARB_files/SSURef_NR99_128_SILVA_07_09_16_opt.arb.gz
gunzip SSURef_NR99_128_SILVA_07_09_16_opt.arb.gz
arb SSURef_NR99_128_SILVA_07_09_16_opt.arb

This will launch us into the arb environment with the ‘‘Ref NR 99’’ database opened. This database has 597,607 sequences within it that are not more than 99% similar to each other. The release notes for this database as well as the idea behind the non-redundant database are available from the silva website. Within arb do the following:

Click the search button
Set the first search field to ‘ARB_color’ and set it to 1. Click on the equal sign until it indicates not equal (this removes low quality reads and chimeras)
Click ‘Search’. This yielded 577,832 hits
Click the “Mark Listed Unmark Rest” button
Close the “Search and Query” box
Now click on File->export->export to external format
In this box the Export option should be set to marked, Filter to none, and Compression should be set to no.
In the field for Choose an output file name enter make sure the path has you in the correct working directory and enter silva.full_v128.fasta.
Select a format: fasta_mothur.eft. This is a custom formatting file that I have created that includes the sequences accession number and it’s taxonomy across the top line. To create one for you will need to create fasta_mothur.eft in the /opt/local/share/arb/lib/export/ folder with the following:
```
SUFFIX          fasta    
BEGIN    
>*(acc).*(name)\t*(align_ident_slv)\t*(tax_slv);    
*(|export_sequence)    
```
Save this as silva.full_v128.fasta
You can now quit arb.

Screening the sequences

mothur "#screen.seqs(fasta=silva.full_v128.fasta, start=1044, end=43116, maxambig=5, processors=8);
        pcr.seqs(start=1044, end=43116, keepdots=T);
        degap.seqs();
        unique.seqs();"

#identify the unique sequences without regard to their alignment
grep ">" silva.full_v128.good.pcr.ng.unique.fasta | cut -f 1 | cut -c 2- > silva.full_v128.good.pcr.ng.unique.accnos

#get the unique sequences without regard to their alignment
mothur "#get.seqs(fasta=silva.full_v128.good.pcr.fasta, accnos=silva.full_v128.good.pcr.ng.unique.accnos)"

#generate alignment file
mv silva.full_v128.good.pcr.pick.fasta silva.nr_v128.align

#generate taxonomy file
grep '>' silva.nr_v128.align | cut -f1,3 | cut -f2 -d'>' > silva.nr_v128.full

The mothur commands above do several things. First the screen.seqs command removes sequences that are not full length and have more than 5 ambiguous base calls. Note: this will remove a number of Archaea since the ARB RN reference database lets in shorter (>900 bp) archaeal 16S rRNA gene sequences. Second, pcr.seqs convert any base calls that occur before position 1044 and after 43116 to . to make them only span the region between the 27f and 1492r priming sites. Finally, it is possible that weird things happen in the alignments and so we unalign the sequences (degap.seqs) and identify the unique sequences (unique.seqs). We then convert the resulting fasta file into an accnos file so that we can go back into mothur and pull out the unique sequences from the aligned file (get.seqs).

Formatting the taxonomy files

Now we want to make sure the taxonomy file is properly formatted for use with mothur. First we want to grab the SILVA taxa mapping file by running the following in bash:

wget https://www.arb-silva.de/fileadmin/silva_databases/current/Exports/taxonomy/tax_slv_ssu_128.txt

map.in <- read.table("tax_slv_ssu_128.txt",header=F,sep="\t",stringsAsFactors=F)
map.in <- map.in[,c(1,3)]
colnames(map.in) <- c("taxlabel","taxlevel")
map.in <- rbind(map.in, c("Bacteria;RsaHf231;", "phylum")) #wasn't in tax_slv_ssu_128.txt

#fix Escherichia nonsense
map.in$taxlevel[which(map.in$taxlabel=="Bacteria;Proteobacteria;Gammaproteobacteria;Enterobacteriales;Enterobacteriaceae;Escherichia;")] <- "genus"

taxlevels <- c("root","domain","major_clade","superkingdom","kingdom","subkingdom","infrakingdom","superphylum","phylum","subphylum","infraphylum","superclass","class","subclass","infraclass","superorder","order","suborder","superfamily","family","subfamily","genus")
taxabb <- c("ro","do","mc","pk","ki","bk","ik","pp","ph","bp","ip","pc","cl","bc","ic","po","or","bo","pf","fa","bf","ge")
tax.mat <- matrix(data="",nrow=nrow(map.in),ncol=length(taxlevels))
tax.mat[,1] <- "root"
colnames(tax.mat) <- taxlevels

outlevels <- c("domain","phylum","class","order","family","genus")

for (i in 1:nrow(map.in)) {
	taxname <- unlist(strsplit(as.character(map.in[i,1]), split=';'))
	#print(taxname);

	while ( length(taxname) > 0) {
		#regex to look for exact match

		tax.exp <- paste(paste(taxname,collapse=";"),";",sep="")
		tax.match <- match(tax.exp,map.in$taxlabel)
		tax.mat[i,map.in[tax.match,2]] <- tail(taxname,1)
		taxname <- head(taxname,-1)
	}
}

for (i in 1:nrow(tax.mat)) {
	#this fills in the empty gaps by using the closest higher taxonomic level appended with an abbreviation for the current taxonomic level
	#if you don't want this behavior, cut it out
	for (j in 1:ncol(tax.mat)) {
		if(tax.mat[i,j] < 0) { tax.mat[i,j] <- paste(tmptax,taxabb[j],sep="_")}
		else { tmptax <- tax.mat[i,j]}
	}

	#this maps the new name to the input taxonomic levels
	map.in[i,"taxout"] <- paste(paste(tax.mat[i,outlevels],collapse=";"),";",sep="")
}

# replace spaces with underscores
map.in$taxout <- gsub(" ","_",map.in$taxout)

# bring in the old taxonomic levels from SILVA and remap them using the new levels
tax.in <- read.table("silva.nr_v128.full",header=F,stringsAsFactors=F,sep="\t")
colnames(tax.in) <- c("taxid","taxlabel")

tax.in$taxlabel <- gsub("[[:space:]]+;", ";", tax.in$taxlabel) #fix extra space in "Eukaryota;Opisthokonta;Nucletmycea;Fungi;Dikarya;Ascomycota;Pezizomycotina;Dothideomycetes;Pleosporales;Phaeosphaeriaceae;Parastagonospora ;"
tax.in$taxlabel <- gsub(";[[:space:]]+$", ";", tax.in$taxlabel)

tax.in$id <- 1:nrow(tax.in)

tax.write <- merge(tax.in,map.in,all.x=T,sort=F)
tax.write <- tax.write[order(tax.write$id),]


#we want to see whether everything has 6 taxonomic level (kingdom to genus)
getDepth <- function(taxonString){
  initial <- nchar(taxonString)
    removed <- nchar(gsub(";", "", taxonString))
    return(initial-removed)
}

depth <- getDepth(tax.write$taxout)
summary(depth) #should all be 6
bacteria <- grepl("Bacteria;", tax.write$taxout)
archaea <- grepl("Archaea;", tax.write$taxout)
eukarya <- grepl("Eukaryota;", tax.write$taxout)

tax.write[depth > 6 & bacteria,] #good to go
tax.write[depth > 6 & archaea,]  #good to go
tax.write[depth > 6 & eukarya,]  #good to go

write.table(tax.write[,c("taxid","taxout")],file="silva.full_v128.tax",sep="\t",row.names=F,quote=F,col.names=F)

Building the SEED references

The first thing to note is that SILVA does not release their SEED; it is private. By screening through the ARB databases we can attempt to recreate it. Our previous publications show that classify.seqs with the recreated SEED does an excellent job of realigning sequences to look like they would if you used SINA and the true SEED. Now we want to try to figure out which sequences are part of the seed. Earlier, when we exported the sequences from ARB, we included the align_ident_slv field from the database in our output. Let’s generate an accnos file that contains the names of the sequences with 100% to the SEED database and then use mothur to generate SEED fasta and taxonomy files. While we’re at it we’ll also generate the nr_128 taxonomy file as well. The following code will be run from within a bash terminal:

grep ">" silva.nr_v128.align | cut -f 1,2 | grep "\t100" | cut -f 1 | cut -c 2- > silva.seed_v128.accnos
mothur "#get.seqs(fasta=silva.nr_v128.align, taxonomy=silva.full_v128.tax, accnos=silva.seed_v128.accnos)"
mv silva.nr_v128.pick.align silva.seed_v128.align
mv silva.full_v128.pick.tax silva.seed_v128.tax

mothur "#get.seqs(taxonomy=silva.full_v128.tax, accnos=silva.full_v128.good.pcr.ng.unique.accnos)"
mv silva.full_v128.pick.tax silva.nr_v128.tax

Taxonomic representation

Let’s look to see how many different taxa we have for each taxonomic level within the silva.nr_v128.tax, silva.seed_v128.tax. To do this we’ll run the following in R:

getNumTaxaNames <- function(file, kingdom){
  taxonomy <- read.table(file=file, row.names=1)
  sub.tax <- as.character(taxonomy[grepl(kingdom, taxonomy[,1]),])

  phyla <- as.vector(levels(as.factor(gsub("[^;]*;([^;]*;).*", "\\1", sub.tax))))
  phyla <- sum(!grepl(kingdom, phyla))

  class <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  class <- sum(!grepl(kingdom, class))

  order <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  order <- sum(!grepl(kingdom, order))

  family <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  family <- sum(!grepl(kingdom, family))

  genus <- as.vector(levels(as.factor(gsub("[^;]*;[^;]*;[^;]*;[^;]*;[^;]*;([^;]*;).*", "\\1", sub.tax))))
  genus <- sum(!grepl(kingdom, genus))

  n.seqs <- length(sub.tax)
  return(c(phyla=phyla, class=class, order=order, family=family, genus=genus, n.seqs=n.seqs))
}

kingdoms <- c("Bacteria", "Archaea", "Eukaryota")
tax.levels <- c("phyla", "class", "order", "family", "genus", "n.seqs")

nr.file <- "silva.nr_v128.tax"
nr.matrix <- matrix(rep(0,18), nrow=3)
nr.matrix[1,] <- getNumTaxaNames(nr.file, kingdoms[1])
nr.matrix[2,] <- getNumTaxaNames(nr.file, kingdoms[2])
nr.matrix[3,] <- getNumTaxaNames(nr.file, kingdoms[3])
rownames(nr.matrix) <- kingdoms
colnames(nr.matrix) <- tax.levels
nr.matrix
#          phyla class order family genus n.seqs
#Bacteria     74   261   500   1001  3478 168111
#Archaea      24    52    59    101   217   4337
#Eukaryota   102   252   654    912  2673  18213

seed.file <- "silva.seed_v128.tax"
seed.matrix <- matrix(rep(0,18), nrow=3)
seed.matrix[1,] <- getNumTaxaNames(seed.file, kingdoms[1])
seed.matrix[2,] <- getNumTaxaNames(seed.file, kingdoms[2])
seed.matrix[3,] <- getNumTaxaNames(seed.file, kingdoms[3])
rownames(seed.matrix) <- kingdoms
colnames(seed.matrix) <- tax.levels
seed.matrix
#          phyla class order family genus n.seqs
#Bacteria     54   146   252    471  1375   8512
#Archaea       9    17    24     37    62    147
#Eukaryota    38    96   273    465   957   2554

seed.matrix / nr.matrix
#              phyla     class     order    family     genus     n.seqs
#Bacteria  0.7297297 0.5593870 0.5040000 0.4705295 0.3953422 0.05063321
#Archaea   0.3750000 0.3269231 0.4067797 0.3663366 0.2857143 0.03389440
#Eukaryota 0.3725490 0.3809524 0.4174312 0.5098684 0.3580247 0.14022951

Finally, we want to compress the resulting alignment and this README file into the full length and SEED archives using commands in the bash terminal:

tar cvzf silva.nr_v128.tgz silva.nr_v128.tax silva.nr_v128.align README.*
tar cvzf silva.seed_v128.tgz silva.seed_v128.tax silva.seed_v128.align README.*

Application

So… which to use for what application? If you have the RAM, I’d suggest using silva.nr_v128.align in align.seqs. It took about 10 minutes to read in the database file and a minute or so to align a 1000 full-length sequences. Here is an example workflow for use within mothur that will get you the V4 region of the 16S rRNA gene:

mothur "#pcr.seqs(fasta=silva.nr_v128.align, start=11894, end=25319, keepdots=F, processors=8);
        unique.seqs()"

This will get you 104,711 unique sequences to then align against (meh.). Other tricks to consider would be to use get.lineage to pull out the reference sequences that are from the Bacteria, this will probably only reduce the size of the database by ~10%. You could also try using filter.seqs with vertical=T; however, that might be problematic if there are insertions in your sequences (can’t know a priori). It’s likely that you can just use the silva.seed_v128.align reference for aligning. For classifying sequences, I would strongly recommend using the silva.nr_v128.align and silva.nr_v128.tax references after running pcr.seqs on silva.nr_v128.align. I probably wouldn’t advise using unique.seqs on the output.

Legalese

If you are going to use the files generated in this README, you should be aware of SILVA’s dual use license. We’ll leave it to you to work out the details.

README for the RDP v16 reference files

Wed, 15 Mar 2017 00:00:00 +0000

The release notes indicate the following:

RDP Release 11.5 consists of 3,356,809 aligned and annotated 16S rRNA sequences and 125,525 Fungal 28S rRNA sequences. The Bacteria and Archaea hierarchy model used by RDP Classifier and RDP Hierarchy Browser have been updated to training set No. 16. This new training set has over 300 new genera and 2000 new sequences added. There are some rearrangements in genera Gp1, Gp3 and Gp4 of the Acidobacteria due to addition of recently proposed new genera.

Let’s get going…

rm -rf RDPClassifier_16S_trainsetNo16_rawtrainingdata*

wget -N https://sourceforge.net/projects/rdp-classifier/files/RDP_Classifier_TrainingData/RDPClassifier_16S_trainsetNo16_rawtrainingdata.zip
unzip -o RDPClassifier_16S_trainsetNo16_rawtrainingdata.zip
mv RDPClassifier_16S_trainsetNo16_rawtrainingdata/* ./

Now we’d like to start to form the taxonomy file and the fasta file that will be our reference. Again, using bash commands…

mv trainset16_022016.fa trainset16_022016.rdp.fasta
grep ">" trainset16_022016.rdp.fasta | cut -c 2- > trainset16_022016_rmdup.tax

Next, we’d like to get our taxonomy file properly formatted. First we’ll read in the taxonomy data. The following steps are done in R…

tax_file <- scan(file="trainset16_022016_rmdup.tax", what="", sep="\n", quiet=TRUE)

accession <- gsub("^(\\S*).*", "\\1", tax_file) #some are separated by tabs or spaces or both

taxonomy <- gsub(".*(Root.*)", "\\1", tax_file)
taxonomy <- gsub(" ", "_", taxonomy)	#remove spaces and replace with '_'
taxonomy <- gsub("\t", "", taxonomy)	#remove extra tab characters
taxonomy <- gsub("[^;]*_incertae_sedis$", "", taxonomy)
taxonomy <- gsub('\"', '', taxonomy) #remove quote marks

The RDP inserts a variety of sub taxonomic levels (e.g. suborder) that will get in the way of us having a consistent number of taxonomic levels for our analyses. Let’s use the data in trainset16_db_taxid.txt to remove these extra taxonomic levels:

levels <- read.table(file="trainset16_db_taxid.txt", sep="*", stringsAsFactors=FALSE)
subs <- levels[grep("sub", levels$V5),]
sub.names <- subs$V2

tax.split <- strsplit(taxonomy, split=";")

remove.subs <- function(tax.vector){
	return(tax.vector[which(!tax.vector %in% sub.names)])
}

no.subs <- lapply(tax.split, remove.subs)
no.subs.str <- unlist(lapply(no.subs, paste, collapse=";"))
no.subs.str <- gsub("^Root;(.*)$", "\\1;", no.subs.str)

Finally, we can output the taxonomy data to a file we’ll call trainset16_022016.rdp.tax to have a consistent naming scheme with previous versions of those files:

write.table(cbind(as.character(accession), no.subs.str), "trainset16_022016.rdp.tax", row.names=F, col.names=F, quote=F, sep="\t")

wget -N https://mothur.org/w/images/2/24/Trainset10_082014.pds.tgz
tar xvzf Trainset10_082014.pds.tgz
mv trainset10_082014.pds/trainset10_082014* ./
rm -rf trainset10_082014.pds Trainset10_082014.pds.tgz

Now let’s run a mothur command to pull out the extra sequences that are in the pds files:

mothur "#get.lineage(fasta=trainset10_082014.pds.fasta, taxonomy=trainset10_082014.pds.tax, taxon=Eukaryota-Mitochondria)"

This last command gets us the extra “pds” sequences that we can now use to paste on to the end of the normal RDP training set

cat trainset16_022016.rdp.tax trainset10_082014.pds.pick.tax > trainset16_022016.pds.tax
cat trainset16_022016.rdp.fasta trainset10_082014.pds.pick.fasta > trainset16_022016.pds.fasta

wc -l *.pds.tax

##    10773 trainset10_082014.pds.tax
##    13335 trainset16_022016.pds.tax
##    24108 total

Now we’re ready to compress the taxonomy files. First we do the RDP files…

mkdir trainset16_022016.rdp
cp README.* trainset16_022016.rdp.fasta trainset16_022016.rdp.tax trainset16_022016.rdp
tar cvzf trainset16_022016.rdp.tgz  trainset16_022016.rdp/*

a trainset16_022016.rdp/README.md
a trainset16_022016.rdp/trainset16_022016.rdp.fasta
a trainset16_022016.rdp/trainset16_022016.rdp.tax

… and then the pds files…

mkdir trainset16_022016.pds
cp README.* trainset16_022016.pds.fasta trainset16_022016.pds.tax trainset16_022016.pds
tar cvzf trainset16_022016.pds.tgz  trainset16_022016.pds/*

a trainset16_022016.pds/README.md
a trainset16_022016.pds/trainset16_022016.pds.fasta
a trainset16_022016.pds/trainset16_022016.pds.tax