Original Authors: Belinda Phipson, Anna Trigos, Matt Ritchie, Maria Doyle, Harriet Dashnow, Charity Law, Stephane Ballereau, Oscar Rueda, Ashley Sawle Based on the course RNAseq analysis in R delivered on May 11/12th 2016

Resources and data files

This material has been created using the following resources:
http://www.statsci.org/smyth/pubs/QLedgeRPreprint.pdf [@Lun2016]
http://monashbioinformaticsplatform.github.io/RNAseq-DE-analysis-with-R/99-RNAseq_DE_analysis_with_R.html
http://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html https://bioconductor.github.io/BiocWorkshops/rna-seq-data-analysis-with-deseq2.html

Before starting this section, we will make sure we have all the relevant objects from the Differential Expression analysis present.

suppressPackageStartupMessages(library(DESeq2))
load("Robjects/DE.Rdata")
load("Robjects/preprocessing.Rdata")

Overview

  • Visualising DE results
  • Getting annotation using Bioconductor databases
  • Getting annotation using BiomaRt
  • Customising RNA-seq plots with ggplot2
  • Retrieving gene models

We can now have a list of genes ordered according to their evidence for being differentially-expressed.

library(dplyr)
library(tibble)
results.status <- as.data.frame(results(de.mf,contrast=c("Status","lactation","virgin"))) %>%
  rownames_to_column("ENSEMBL")
  
results.ordered <- arrange(results.status, padj)
head(results.ordered)
             ENSEMBL   baseMean log2FoldChange     lfcSE     stat
1 ENSMUSG00000000381  398943.44       9.779005 0.4261732 22.94608
2 ENSMUSG00000061937 1134857.22       8.204288 0.4158950 19.72683
3 ENSMUSG00000026417   11320.43       6.589672 0.3634068 18.13304
4 ENSMUSG00000022491  514850.04       9.589474 0.5336395 17.96995
5 ENSMUSG00000061388   30555.67      10.172377 0.5858320 17.36398
6 ENSMUSG00000024903   60039.52       8.772293 0.5388825 16.27868
         pvalue          padj
1 1.612341e-116 2.785481e-112
2  1.268851e-86  1.096033e-82
3  1.748099e-73  1.006672e-69
4  3.350312e-72  1.447000e-68
5  1.546170e-67  5.342325e-64
6  1.398617e-59  4.027083e-56

In DESeq2, the function plotMA shows the log2 fold changes attributable to a given variable over the mean of normalized counts for all the samples in the DESeqDataSet. Points will be colored red if the adjusted p value is less than 0.1. Points which fall out of the window are plotted as open triangles pointing either up or down.

The log2 fold change for a particular comparison is plotted on the y-axis and the average of the counts normalized by size factor is shown on the x-axis (“M” for minus, because a log ratio is equal to log minus log, and “A” for average). Each gene is represented with a dot. Genes with an adjusted p value below a threshold (here 0.1, the default) are shown in red.

plotMA(results(de.mf,contrast=c("Status","lactation","virgin")))

Note You may see an error message when trying to make the above MA plot. This could be because both limma and DESeq2 have a function called plotMA, and R can sometimes pick the wrong function. To explictly use the DESeq2 function you can use:-

DESeq2::plotMA(results(de.mf,contrast=c("Status","lactation","virgin")))

MA-plots often display a fanning-effect at the left-hand side (genes with low numbers of counts) due to the high variability of the measurements for these genes. For more informative visualization and more accurate ranking of genes by effect size (the log fold change may sometimes be referred to as an effect size), the DESeq2 authors recommend “shrinking” the log fold-changes which is available in DESeq2’s lfcShrink function. This results in more stable fold change values. The p-values are unaffected.

res_LvsV <- lfcShrink(de.mf,contrast=c("Status","lactation","virgin"))
DESeq2::plotMA(res_LvsV)

We will re-define our results object to use these new fold-changes.

results.ordered <- as.data.frame(res_LvsV) %>% 
  rownames_to_column("ENSEMBL") %>% 
  arrange(padj)
head(results.ordered)
             ENSEMBL   baseMean log2FoldChange     lfcSE     stat
1 ENSMUSG00000000381  398943.44       8.722325 0.3815879 22.94608
2 ENSMUSG00000061937 1134857.22       7.351405 0.3761569 19.72683
3 ENSMUSG00000026417   11320.43       6.078220 0.3332017 18.13304
4 ENSMUSG00000022491  514850.04       7.765956 0.4539439 17.96995
5 ENSMUSG00000061388   30555.67       8.282907 0.4740455 17.36398
6 ENSMUSG00000024903   60039.52       7.312158 0.4522371 16.27868
         pvalue          padj
1 1.612341e-116 2.785481e-112
2  1.268851e-86  1.096033e-82
3  1.748099e-73  1.006672e-69
4  3.350312e-72  1.447000e-68
5  1.546170e-67  5.342325e-64
6  1.398617e-59  4.027083e-56

Another common plot for displaying the results of a differential expression analysis is a volcano plot

library(ggplot2)
results.ordered %>% 
  ggplot(aes(x = log2FoldChange, y = -log10(padj))) + geom_point()

It can also be useful to examine the counts of reads for a single gene across the groups. A simple function for making this plot is plotCounts, which normalizes counts by sequencing depth and adds a pseudocount of 1/2 to allow for log scale plotting. The counts are grouped by the variables in intgroup, where more than one variable can be specified. Here we specify the gene which had the smallest p value from the results table created above. You can select the gene to plot by rowname or by numeric index:-

plotCounts(dds, "ENSMUSG00000000381",intgroup = c("Status"))

If we want greater control over how to visualise the data, we can use the plotCounts function to return the count data, but not actually produce the plot:-

plotCounts(dds, "ENSMUSG00000000381",intgroup = c("Status"),returnData=TRUE)
                  count    Status
SRR1552444 1.569482e+03    virgin
SRR1552445 1.671109e+03    virgin
SRR1552446 9.802440e+04 pregnancy
SRR1552447 2.205304e+05 pregnancy
SRR1552448 2.292659e+06 lactation
SRR1552449 2.137836e+06 lactation
SRR1552450 2.427758e+01    virgin
SRR1552451 3.357483e+01    virgin
SRR1552452 3.410069e+03 pregnancy
SRR1552453 4.660197e+03 pregnancy
SRR1552454 1.368587e+04 lactation
SRR1552455 1.322326e+04 lactation

Challenge 1

  1. Use the option returnData=TRUE to get a data frame containing the counts of ENSMUSG00000000381 in the different development stages. Visualise these data using ggplot2 (see plot A below).
  2. Repeat the volcano plot from above, but use a different colour to indicate which genes are significant with an adjusted p-value less than 0.05. See plot B below
  3. (Optional) The argument intgroup= can be used to retrieve and plot data from multiple variables of interest in the data. Use the value intgroup=c("Status","CellType") and compare the counts between different cell types and status. See plot C below. HINT: To get the counts on the same scale as displayed by the plotCounts function you will need to add +scale_y_log10 in your ggplot2 code

However, it is hard to assess the biological significance of such a gene without more information about . To perform such a task we need to map between the identifiers we have in the DESeq2 output and more familiar names.

Adding annotation to the DESeq2 results

There are a number of ways to add annotation, but we will demonstrate how to do this using the org.Mm.eg.db package. This package is one of several organism-level packages which are re-built every 6 months. These packages are listed on the annotation section of the Bioconductor, and are installed in the same way as regular Bioconductor packages. An alternative approach is to use biomaRt, an interface to the BioMart resource. BioMart is much more comprehensive, but the organism packages fit better into the Bioconductor workflow.

### Only execute when you need to install the package
install.packages("BiocManager")
BiocManager::install("org.Mm.eg.db")
# For Human
BiocManager::install("org.Hs.eg.db")

The packages are larger in size that Bioconductor software pacakges, but essentially they are databases that can be used to make offline queries.

library(org.Mm.eg.db)

First we need to decide what information we want. In order to see what we can extract we can run the columns function on the annotation database.

columns(org.Mm.eg.db)
 [1] "ACCNUM"       "ALIAS"        "ENSEMBL"      "ENSEMBLPROT" 
 [5] "ENSEMBLTRANS" "ENTREZID"     "ENZYME"       "EVIDENCE"    
 [9] "EVIDENCEALL"  "GENENAME"     "GO"           "GOALL"       
[13] "IPI"          "MGI"          "ONTOLOGY"     "ONTOLOGYALL" 
[17] "PATH"         "PFAM"         "PMID"         "PROSITE"     
[21] "REFSEQ"       "SYMBOL"       "UNIGENE"      "UNIPROT"     

We are going to filter the database by a key or set of keys in order to extract the information we want. Valid names for the key can be retrieved with the keytypes function.

keytypes(org.Mm.eg.db)
 [1] "ACCNUM"       "ALIAS"        "ENSEMBL"      "ENSEMBLPROT" 
 [5] "ENSEMBLTRANS" "ENTREZID"     "ENZYME"       "EVIDENCE"    
 [9] "EVIDENCEALL"  "GENENAME"     "GO"           "GOALL"       
[13] "IPI"          "MGI"          "ONTOLOGY"     "ONTOLOGYALL" 
[17] "PATH"         "PFAM"         "PMID"         "PROSITE"     
[21] "REFSEQ"       "SYMBOL"       "UNIGENE"      "UNIPROT"     

We should see ENSEMBL, which is the type of key we are going to use in this case. If we are unsure what values are acceptable for the key, we can check what keys are valid with keys

keys(org.Mm.eg.db, keytype="ENSEMBL")[1:10]
 [1] "ENSMUSG00000030359" "ENSMUSG00000020804" "ENSMUSG00000025375"
 [4] "ENSMUSG00000015243" "ENSMUSG00000028125" "ENSMUSG00000026944"
 [7] "ENSMUSG00000031333" "ENSMUSG00000024030" "ENSMUSG00000058835"
[10] "ENSMUSG00000026842"

For the top gene in our analysis the call to the function would be:-

select(org.Mm.eg.db, keys="ENSMUSG00000000381",
       keytype = "ENSEMBL",columns=c("SYMBOL","GENENAME")
)

Unfortunately, the authors of dplyr and AnnotationDbi have both decided to use the name select in their packages. To avoid confusion, the following code is sometimes used:-

AnnotationDbi::select(org.Mm.eg.db, keys="ENSMUSG00000000381",keytype = "ENSEMBL",columns=c("SYMBOL","GENENAME"))
             ENSEMBL SYMBOL            GENENAME
1 ENSMUSG00000000381    Wap whey acidic protein

To annotate our results, we definitely want gene symbols and perhaps the full gene name. Let’s build up our annotation information into a new data frame using the select function.

anno <- AnnotationDbi::select(org.Mm.eg.db,keys=results.ordered$ENSEMBL,
              columns=c("SYMBOL","GENENAME"),
              keytype="ENSEMBL")
# Have a look at the annotation
head(anno)
             ENSEMBL  SYMBOL
1 ENSMUSG00000000381     Wap
2 ENSMUSG00000061937 Csn1s2a
3 ENSMUSG00000026417    Pigr
4 ENSMUSG00000022491 Glycam1
5 ENSMUSG00000061388 Csn1s2b
6 ENSMUSG00000024903    Lao1
                                          GENENAME
1                              whey acidic protein
2                           casein alpha s2-like A
3                polymeric immunoglobulin receptor
4 glycosylation dependent cell adhesion molecule 1
5                           casein alpha s2-like B
6                           L-amino acid oxidase 1

However, we have a problem that the resulting data frame has more rows than our results table. This is due to the one-to-many relationships that often occur when mapping between various identifiers.

dim(anno)
[1] 18071     3
dim(results.ordered)
[1] 17973     7

Such duplicated entries can be identified using the duplicated function.

dup_ids <- anno$ENSEMBL[duplicated(anno$ENSEMBL)]
filter(anno, ENSEMBL %in% dup_ids) %>% 
  arrange(ENSEMBL) %>% head
             ENSEMBL   SYMBOL
1 ENSMUSG00000000486    Sept1
2 ENSMUSG00000000486   Gm4532
3 ENSMUSG00000000562   Adora3
4 ENSMUSG00000000562   Tmigd3
5 ENSMUSG00000001768     Rin2
6 ENSMUSG00000001768 BC039771
                                              GENENAME
1                                             septin 1
2                                  predicted gene 4532
3                                adenosine A3 receptor
4 transmembrane and immunoglobulin domain containing 3
5                             Ras and Rab interactor 2
6                               cDNA sequence BC039771

Fortunately, there are not too many so hopefully we won’t lose too much information if we discard the entries that are duplicated. The first occurence of the duplicated ID will still be included in the table.

anno <- AnnotationDbi::select(org.Mm.eg.db,keys=results.ordered$ENSEMBL,
              columns=c("ENSEMBL","SYMBOL","GENENAME","ENTREZID"),
              keytype="ENSEMBL") %>% 
  filter(!duplicated(ENSEMBL))
dim(anno)
[1] 17973     4

We can bind in the annotation information to the results data frame.

results.annotated <- left_join(results.ordered, anno,by="ENSEMBL")
head(results.annotated)
             ENSEMBL   baseMean log2FoldChange     lfcSE     stat
1 ENSMUSG00000000381  398943.44       8.722325 0.3815879 22.94608
2 ENSMUSG00000061937 1134857.22       7.351405 0.3761569 19.72683
3 ENSMUSG00000026417   11320.43       6.078220 0.3332017 18.13304
4 ENSMUSG00000022491  514850.04       7.765956 0.4539439 17.96995
5 ENSMUSG00000061388   30555.67       8.282907 0.4740455 17.36398
6 ENSMUSG00000024903   60039.52       7.312158 0.4522371 16.27868
         pvalue          padj  SYMBOL
1 1.612341e-116 2.785481e-112     Wap
2  1.268851e-86  1.096033e-82 Csn1s2a
3  1.748099e-73  1.006672e-69    Pigr
4  3.350312e-72  1.447000e-68 Glycam1
5  1.546170e-67  5.342325e-64 Csn1s2b
6  1.398617e-59  4.027083e-56    Lao1
                                          GENENAME ENTREZID
1                              whey acidic protein    22373
2                           casein alpha s2-like A    12993
3                polymeric immunoglobulin receptor    18703
4 glycosylation dependent cell adhesion molecule 1    14663
5                           casein alpha s2-like B    12992
6                           L-amino acid oxidase 1   100470

We can save the results table using the write.csv function, which writes the results out to a csv file that you can open in excel.

write.csv(results.annotated,file="virgin_vs_lactation_DESeq_annotated.csv",row.names=FALSE)

We have already seen the use of a heatmap as a quality assessment tool to visualise the relationship between samples in an experiment. Another common use-case for such a plot is to visualise the results of a differential expression analysis.

Here we will take the top 10 genes from the differential expression analysis and produce a heatmap with the pheatmap package. The default colour palette goes from low expression in blue to high expression in red, which is a good alternative to the traditional red/green heatmaps which are not suitable for those with forms of colour-blindness.

The counts we are visualising are the variance-stablised counts, which are more appropriate for visualisation.

library(pheatmap)
top_genes <- results.annotated$ENSEMBL[1:10]
vsd <- vst(dds)
pheatmap(assay(vsd)[top_genes,])

The heatmap is more informative if we add colours underneath the sample dendrogram to indicate which sample group each sample belongs to. This we can do by creating a data frame containing metadata for each of the samples in our dataset. With the DESeq2 workflow we have already created such a data frame. We have to make sure the the rownames of the data frame are the same as the column names of the counts matrix.

sampleInfo <- as.data.frame(colData(dds)[,c("Status","CellType")])
pheatmap(assay(vsd)[top_genes,],
         annotation_col = sampleInfo)

Any plot we create in RStudio can be saved as a png or pdf file. We use the png or pdf function to create a file for the plot to be saved into and run the rest of the code as normal. The plot does not get displayed in RStudio, but printed to the specified file.

png("heatmap_top10_genes.png",width=800,height=800)
pheatmap(assay(vsd)[top_genes,],
         annotation_col = sampleInfo)
# dev.off()

Challenge 2

  1. Repeat the same heatmap as above, but for the top 100 most differentially-expressed genes between pregnant and luminal
  2. Change the plot so that gene names are displayed rather than Ensembl IDs
  3. Save the plot to a pdf file HINT: check the help for the pheatmap function to see how column and row labels can be changed

Accessing the sample or gene clusters

The heatmap displays relationships between samples and genes in our study as a useful visualisation. In this example we can easily identify which samples are most similar based on their expression patterns. However, for larger dataset this may be more problematic. We can extract data the sample relationships about if we manually perform the clustering steps used by pheatmap. First is to cluster the samples with the default distance matrix and clustering algorithms.

mat <- assay(vsd)[top_genes,]
## Calculate the distance matrix between samples
d_samples <- dist(t(mat))
plot(hclust(d_samples))
rect.hclust(hclust(d_samples),k=2)

We can then “cut” the dendrogram to give a set number of clusters. Each sample has been assigned a label of 1 or 2 depending on which cluster it belongs to.

clusters <- cutree(hclust(d_samples),k = 2)
clusters
SRR1552444 SRR1552445 SRR1552446 SRR1552447 SRR1552448 SRR1552449 
         1          1          2          2          2          2 
SRR1552450 SRR1552451 SRR1552452 SRR1552453 SRR1552454 SRR1552455 
         1          1          1          1          1          1 

The groupings could then be tabulated against with the sample metadata to see if particular biological groups are associated with the new clusters we have identified.

table(clusters, colData(dds)$Status)
        
clusters lactation pregnancy virgin
       1         2         2      4
       2         2         2      0

Adding gene names to a volcano plot

Now that we have an annotated table of results, we can add the gene names to some of the other plots we have created. This should be straightforward as ggplot2 has a label aesthetic that can be mapped to columns in a data frame. The geom_text plot will then display the labels. However, the following plot is a bit crowded.

## Not a good idea to run this!!
results.annotated %>% 
  ggplot(aes(x = log2FoldChange, y = -log10(padj), label=SYMBOL)) + geom_point() + geom_text()

The problem here is that ggplot2 is trying to label every point with a name; not quite what we want. The trick is to create a label that is blank for most genes and only labels the points we are interested in. The ifelse function in R is a convenient way to set the entries in a vector based on a logical expression. In this case, make the values in Label the same as the gene symbol if the gene is in our list of “top genes”. Otherwise, points get labeled with a blank string "".

For clarity, we also make the points slightly transparent and use a different colour for the text.

N <- 10
top_genes <- results.annotated$ENSEMBL[1:N]
results.annotated %>% 
  mutate(Label = ifelse(ENSEMBL %in% top_genes, SYMBOL, "")) %>%  
  ggplot(aes(x = log2FoldChange, y = -log10(padj), label=Label)) + geom_point(alpha=0.4) + geom_text(col="blue")

Finally, a slightly better positioning of text is given by the ggrepel package.

if(!require(ggrepel)) install.packages("ggrepel")
results.annotated %>% 
  mutate(Label = ifelse(ENSEMBL %in% top_genes, SYMBOL, "")) %>%  
  ggplot(aes(x = log2FoldChange, y = -log10(padj), label=Label)) + geom_point(alpha=0.4) + geom_text_repel(col="blue")

Annotation with the biomaRt resource

The Bioconductor package have the convenience of being able to make queries offline. However, they are only available for certain organisms. If your organism does not have an org.XX.eg.db package listed on the Bioconductor annotation page (http://bioconductor.org/packages/release/BiocViews.html#___AnnotationData), an alternative is to use biomaRt which provides an interface to the popular biomart annotation resource.

The first step is to find the name of a database that you want to connect to

library(biomaRt)
listMarts()
               biomart               version
1 ENSEMBL_MART_ENSEMBL      Ensembl Genes 99
2   ENSEMBL_MART_MOUSE      Mouse strains 99
3     ENSEMBL_MART_SNP  Ensembl Variation 99
4 ENSEMBL_MART_FUNCGEN Ensembl Regulation 99
ensembl=useMart("ENSEMBL_MART_ENSEMBL")
# list the available datasets (species). Replace mouse with the name of your organism
listDatasets(ensembl) %>% filter(grepl("Mouse",description))
                 dataset                  description   version
1  mmurinus_gene_ensembl Mouse Lemur genes (Mmur_3.0)  Mmur_3.0
2 mmusculus_gene_ensembl      Mouse genes (GRCm38.p6) GRCm38.p6
ensembl = useDataset("mmusculus_gene_ensembl", mart=ensembl)

Queries to biomaRt are constructed in a similar way to the queries we performed with the org.Mm.eg.db package. Instead of keys we have filters, and instead of columns we have attributes. The list of acceptable values is much more comprehensive that for the org.Mm.eg.db package.

listFilters(ensembl) %>% 
    filter(grepl("ensembl",name))
                                  name
1        with_clone_based_ensembl_gene
2  with_clone_based_ensembl_transcript
3                      ensembl_gene_id
4              ensembl_gene_id_version
5                ensembl_transcript_id
6        ensembl_transcript_id_version
7                   ensembl_peptide_id
8           ensembl_peptide_id_version
9                      ensembl_exon_id
10            clone_based_ensembl_gene
11      clone_based_ensembl_transcript
                                                        description
1                             With Clone-based (Ensembl) gene ID(s)
2                       With Clone-based (Ensembl) transcript ID(s)
3                       Gene stable ID(s) [e.g. ENSMUSG00000000001]
4        Gene stable ID(s) with version [e.g. ENSMUSG00000000001.4]
5                 Transcript stable ID(s) [e.g. ENSMUST00000000001]
6  Transcript stable ID(s) with version [e.g. ENSMUST00000000001.4]
7                    Protein stable ID(s) [e.g. ENSMUSP00000000001]
8     Protein stable ID(s) with version [e.g. ENSMUSP00000000001.4]
9                              Exon ID(s) [e.g. ENSMUSE00000097910]
10               Clone-based (Ensembl) gene ID(s) [e.g. AC015535.1]
11     Clone-based (Ensembl) transcript ID(s) [e.g. AC015535.1-201]
listAttributes(ensembl) %>% 
    filter(grepl("gene",name))

An advantage over the org.. packages is that positional information can be retrieved

attributeNames <- c('ensembl_gene_id', 'entrezgene_id', 'external_gene_name')
getBM(attributes = attributeNames,
      filters = "ensembl_gene_id",
      values=top_genes,
      mart=ensembl)
      ensembl_gene_id entrezgene_id external_gene_name
1  ENSMUSG00000000381         22373                Wap
2  ENSMUSG00000022491         14663            Glycam1
3  ENSMUSG00000024331         13506               Dsc2
4  ENSMUSG00000024903        100470               Lao1
5  ENSMUSG00000026417         18703               Pigr
6  ENSMUSG00000040260         76441              Daam2
7  ENSMUSG00000061388         12992            Csn1s2b
8  ENSMUSG00000061937         12993            Csn1s2a
9  ENSMUSG00000063275         30963              Hacd1
10 ENSMUSG00000070702         12990             Csn1s1

Challenge 3

  1. Use biomaRt to create an data frame containing the entrezgene, gene symbol and genomic coordinates (chromosome, start, end) for the Ensembl IDs in the DESeq2 results
  2. Remove duplicates entries from the new data frame
  3. Join the biomaRt annotation to the DESeq2 results to produce a data frame with differential expression results and annotation
  4. Write the joined data frame to a csv file

Obtaining gene models with Bioconductor

Using biomaRt as above allows us to retrieve the genomic coordinates of a given gene. If we want more-comprehensive information about the structue of a gene (and indeed all genes in the transcriptome), we can use one of several pre-built transcript database packages.

Retrieving the coordinates for a particular gene (or set of genes) uses the same select function as for the org.Mm.eg.db package, but checking for different columns and keytypes.

library(TxDb.Mmusculus.UCSC.mm10.knownGene)
txdb <- TxDb.Mmusculus.UCSC.mm10.knownGene
columns(txdb)
 [1] "CDSCHROM"   "CDSEND"     "CDSID"      "CDSNAME"    "CDSPHASE"  
 [6] "CDSSTART"   "CDSSTRAND"  "EXONCHROM"  "EXONEND"    "EXONID"    
[11] "EXONNAME"   "EXONRANK"   "EXONSTART"  "EXONSTRAND" "GENEID"    
[16] "TXCHROM"    "TXEND"      "TXID"       "TXNAME"     "TXSTART"   
[21] "TXSTRAND"   "TXTYPE"    
AnnotationDbi::select(txdb, columns = c("EXONID","EXONSTART","EXONCHROM"),
                      keys="22373", 
                      keytype = "GENEID")
  GENEID EXONID EXONCHROM EXONSTART
1  22373 163805     chr11   6638535
2  22373 163804     chr11   6637190
3  22373 163803     chr11   6636710
4  22373 163802     chr11   6635483

Alternatively we can grab the coordinates of all genes in a single object and then subset accordingly. This allows us to perform all kinds of subset operation using the GenomicFeatures framework in Bioconductor.

exons <- exonsBy(txdb,"gene")
exons
GRangesList object of length 24421:
$100009600 
GRanges object with 7 ranges and 2 metadata columns:
      seqnames            ranges strand |   exon_id   exon_name
         <Rle>         <IRanges>  <Rle> | <integer> <character>
  [1]     chr9 21062393-21062717      - |    134805        <NA>
  [2]     chr9 21062894-21062987      - |    134806        <NA>
  [3]     chr9 21063314-21063396      - |    134807        <NA>
  [4]     chr9 21066024-21066377      - |    134808        <NA>
  [5]     chr9 21066940-21067925      - |    134809        <NA>
  [6]     chr9 21068030-21068117      - |    134810        <NA>
  [7]     chr9 21073075-21075496      - |    134812        <NA>

$100009609 
GRanges object with 6 ranges and 2 metadata columns:
      seqnames            ranges strand | exon_id exon_name
  [1]     chr7 84940169-84941088      - |  110194      <NA>
  [2]     chr7 84943141-84943264      - |  110195      <NA>
  [3]     chr7 84943504-84943722      - |  110196      <NA>
  [4]     chr7 84946200-84947000      - |  110197      <NA>
  [5]     chr7 84947372-84947651      - |  110198      <NA>
  [6]     chr7 84963816-84964009      - |  110199      <NA>

$100009614 
GRanges object with 1 range and 2 metadata columns:
      seqnames            ranges strand | exon_id exon_name
  [1]    chr10 77711446-77712009      + |  144278      <NA>

...
<24418 more elements>
-------
seqinfo: 66 sequences (1 circular) from mm10 genome
exons[["22373"]]
GRanges object with 4 ranges and 2 metadata columns:
      seqnames          ranges strand |   exon_id   exon_name
         <Rle>       <IRanges>  <Rle> | <integer> <character>
  [1]    chr11 6635483-6635629      - |    163802        <NA>
  [2]    chr11 6636710-6636874      - |    163803        <NA>
  [3]    chr11 6637190-6637324      - |    163804        <NA>
  [4]    chr11 6638535-6638649      - |    163805        <NA>
  -------
  seqinfo: 66 sequences (1 circular) from mm10 genome

Interactive graphs and tables

It is often useful to be able to explore our results in an interactive manner; searching for our favourite genes of interest and plotting on-the-fly whether they are statistically significant in our dataset or not.

Such a visualisation is possible with the Glimma Bioconductor package.

It takes our DESeq2 results object, annotation table and normalized counts, and produces a HTML page including a sortable results table, MA-plot and scatter plot. Particular genes can be searched among the table and their expression patterns can be displayed. Alternatively we can click on particular point in the plot and display their stats.

results <- results(de.mf,contrast=c("Status","lactation","virgin"))
## Repeat the annotation, as the previous annotation table was created using an ordered results table
anno <- AnnotationDbi::select(org.Mm.eg.db,keys=rownames(results),
              columns=c("SYMBOL","GENENAME"),
              keytype="ENSEMBL") %>% 
  filter(!duplicated(ENSEMBL))
## Make sure we have normalised counts before proceeding
dds <- estimateSizeFactors(dds)
## Load the Glimma package and create the report
library(Glimma)
glMDPlot(results,
         anno,
         groups = colData(dds)$Status,
         counts = counts(dds,normalized=TRUE),
         transform = TRUE,
         side.main = "ENSEMBL")
LS0tCnRpdGxlOiAiUk5BLXNlcSBBbmFseXNpcyBpbiBSIgpzdWJ0aXRsZTogIkFubm90YXRpb24gYW5kIFZpc3VhbGlzYXRpb24gb2YgUk5BLXNlcSByZXN1bHRzIgphdXRob3I6ICJNYXJrIER1bm5pbmciCmRhdGU6ICJGZWJydWFyeSAyMDIwIgpvdXRwdXQ6CiAgaHRtbF9ub3RlYm9vazoKICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwogIGh0bWxfZG9jdW1lbnQ6CiAgICB0b2M6IHllcwogICAgdG9jX2Zsb2F0OiB5ZXMKbWludXRlczogMzAwCmxheW91dDogcGFnZQoKLS0tCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUsZmlnLndpZHRoID0gMTIsbWVzc2FnZT1GQUxTRSx3YXJuaW5nPUZBTFNFKQpsaWJyYXJ5KGRwbHlyKQoKYGBgCgoqKk9yaWdpbmFsIEF1dGhvcnM6IEJlbGluZGEgUGhpcHNvbiwgQW5uYSBUcmlnb3MsIE1hdHQgUml0Y2hpZSwgTWFyaWEgRG95bGUsIEhhcnJpZXQgRGFzaG5vdywgQ2hhcml0eSBMYXcqKiwgKipTdGVwaGFuZSBCYWxsZXJlYXUsIE9zY2FyIFJ1ZWRhLCBBc2hsZXkgU2F3bGUqKgpCYXNlZCBvbiB0aGUgY291cnNlIFtSTkFzZXEgYW5hbHlzaXMgaW4gUl0oaHR0cDovL2NvbWJpbmUtYXVzdHJhbGlhLmdpdGh1Yi5pby8yMDE2LTA1LTExLVJOQXNlcS8pIGRlbGl2ZXJlZCBvbiBNYXkgMTEvMTJ0aCAyMDE2CgojIyBSZXNvdXJjZXMgYW5kIGRhdGEgZmlsZXMKClRoaXMgbWF0ZXJpYWwgaGFzIGJlZW4gY3JlYXRlZCB1c2luZyB0aGUgZm9sbG93aW5nIHJlc291cmNlczogIApodHRwOi8vd3d3LnN0YXRzY2kub3JnL3NteXRoL3B1YnMvUUxlZGdlUlByZXByaW50LnBkZiBbQEx1bjIwMTZdICAKaHR0cDovL21vbmFzaGJpb2luZm9ybWF0aWNzcGxhdGZvcm0uZ2l0aHViLmlvL1JOQXNlcS1ERS1hbmFseXNpcy13aXRoLVIvOTktUk5Bc2VxX0RFX2FuYWx5c2lzX3dpdGhfUi5odG1sICAKaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sCmh0dHBzOi8vYmlvY29uZHVjdG9yLmdpdGh1Yi5pby9CaW9jV29ya3Nob3BzL3JuYS1zZXEtZGF0YS1hbmFseXNpcy13aXRoLWRlc2VxMi5odG1sCgoKCkJlZm9yZSBzdGFydGluZyB0aGlzIHNlY3Rpb24sIHdlIHdpbGwgbWFrZSBzdXJlIHdlIGhhdmUgYWxsIHRoZSByZWxldmFudCBvYmplY3RzIGZyb20gdGhlIERpZmZlcmVudGlhbCBFeHByZXNzaW9uIGFuYWx5c2lzIHByZXNlbnQuCgpgYGB7cn0Kc3VwcHJlc3NQYWNrYWdlU3RhcnR1cE1lc3NhZ2VzKGxpYnJhcnkoREVTZXEyKSkKCmxvYWQoIlJvYmplY3RzL0RFLlJkYXRhIikKbG9hZCgiUm9iamVjdHMvcHJlcHJvY2Vzc2luZy5SZGF0YSIpCmBgYAoKIyBPdmVydmlldwoKLSBWaXN1YWxpc2luZyBERSByZXN1bHRzCi0gR2V0dGluZyBhbm5vdGF0aW9uIHVzaW5nIEJpb2NvbmR1Y3RvciBkYXRhYmFzZXMKLSBHZXR0aW5nIGFubm90YXRpb24gdXNpbmcgQmlvbWFSdAotIEN1c3RvbWlzaW5nIFJOQS1zZXEgcGxvdHMgd2l0aCBgZ2dwbG90MmAKLSBSZXRyaWV2aW5nIGdlbmUgbW9kZWxzCgoKCgpXZSBjYW4gbm93IGhhdmUgYSBsaXN0IG9mIGdlbmVzIG9yZGVyZWQgYWNjb3JkaW5nIHRvIHRoZWlyIGV2aWRlbmNlIGZvciBiZWluZyBkaWZmZXJlbnRpYWxseS1leHByZXNzZWQuCgpgYGB7cn0KbGlicmFyeShkcGx5cikKbGlicmFyeSh0aWJibGUpCgpyZXN1bHRzLnN0YXR1cyA8LSBhcy5kYXRhLmZyYW1lKHJlc3VsdHMoZGUubWYsY29udHJhc3Q9YygiU3RhdHVzIiwibGFjdGF0aW9uIiwidmlyZ2luIikpKSAlPiUKICByb3duYW1lc190b19jb2x1bW4oIkVOU0VNQkwiKQogIAoKcmVzdWx0cy5vcmRlcmVkIDwtIGFycmFuZ2UocmVzdWx0cy5zdGF0dXMsIHBhZGopCmhlYWQocmVzdWx0cy5vcmRlcmVkKQpgYGAKCkluIGBERVNlcTJgLCB0aGUgZnVuY3Rpb24gcGxvdE1BIHNob3dzIHRoZSBsb2cyIGZvbGQgY2hhbmdlcyBhdHRyaWJ1dGFibGUgdG8gYSBnaXZlbiB2YXJpYWJsZSBvdmVyIHRoZSBtZWFuIG9mIG5vcm1hbGl6ZWQgY291bnRzIGZvciBhbGwgdGhlIHNhbXBsZXMgaW4gdGhlIERFU2VxRGF0YVNldC4gUG9pbnRzIHdpbGwgYmUgY29sb3JlZCByZWQgaWYgdGhlIGFkanVzdGVkIHAgdmFsdWUgaXMgbGVzcyB0aGFuIDAuMS4gUG9pbnRzIHdoaWNoIGZhbGwgb3V0IG9mIHRoZSB3aW5kb3cgYXJlIHBsb3R0ZWQgYXMgb3BlbiB0cmlhbmdsZXMgcG9pbnRpbmcgZWl0aGVyIHVwIG9yIGRvd24uCgpUaGUgbG9nMiBmb2xkIGNoYW5nZSBmb3IgYSBwYXJ0aWN1bGFyIGNvbXBhcmlzb24gaXMgcGxvdHRlZCBvbiB0aGUgeS1heGlzIGFuZCB0aGUgYXZlcmFnZSBvZiB0aGUgY291bnRzIG5vcm1hbGl6ZWQgYnkgc2l6ZSBmYWN0b3IgaXMgc2hvd24gb24gdGhlIHgtYXhpcyAoIk0iIGZvciBtaW51cywgYmVjYXVzZSBhIGxvZyByYXRpbyBpcyBlcXVhbCB0byBsb2cgbWludXMgbG9nLCBhbmQgIkEiIGZvciBhdmVyYWdlKS4gRWFjaCBnZW5lIGlzIHJlcHJlc2VudGVkIHdpdGggYSBkb3QuIEdlbmVzIHdpdGggYW4gYWRqdXN0ZWQgcCB2YWx1ZSBiZWxvdyBhIHRocmVzaG9sZCAoaGVyZSAwLjEsIHRoZSBkZWZhdWx0KSBhcmUgc2hvd24gaW4gcmVkLgoKCmBgYHtyfQpwbG90TUEocmVzdWx0cyhkZS5tZixjb250cmFzdD1jKCJTdGF0dXMiLCJsYWN0YXRpb24iLCJ2aXJnaW4iKSkpCmBgYAoKKioqTm90ZSoqKiBZb3UgbWF5IHNlZSBhbiBlcnJvciBtZXNzYWdlIHdoZW4gdHJ5aW5nIHRvIG1ha2UgdGhlIGFib3ZlIE1BIHBsb3QuIFRoaXMgY291bGQgYmUgYmVjYXVzZSBib3RoIGBsaW1tYWAgYW5kIGBERVNlcTJgIGhhdmUgYSBmdW5jdGlvbiBjYWxsZWQgYHBsb3RNQWAsIGFuZCBSIGNhbiBzb21ldGltZXMgcGljayB0aGUgd3JvbmcgZnVuY3Rpb24uIFRvIGV4cGxpY3RseSB1c2UgdGhlIGBERVNlcTJgIGZ1bmN0aW9uIHlvdSBjYW4gdXNlOi0KCmBgYHtyfQpERVNlcTI6OnBsb3RNQShyZXN1bHRzKGRlLm1mLGNvbnRyYXN0PWMoIlN0YXR1cyIsImxhY3RhdGlvbiIsInZpcmdpbiIpKSkKYGBgCgpNQS1wbG90cyBvZnRlbiBkaXNwbGF5IGEgZmFubmluZy1lZmZlY3QgYXQgdGhlIGxlZnQtaGFuZCBzaWRlIChnZW5lcyB3aXRoIGxvdyBudW1iZXJzIG9mIGNvdW50cykgZHVlIHRvIHRoZSBoaWdoIHZhcmlhYmlsaXR5IG9mIHRoZSBtZWFzdXJlbWVudHMgZm9yIHRoZXNlIGdlbmVzLiBGb3IgbW9yZSBpbmZvcm1hdGl2ZSB2aXN1YWxpemF0aW9uIGFuZCBtb3JlIGFjY3VyYXRlIHJhbmtpbmcgb2YgZ2VuZXMgYnkgZWZmZWN0IHNpemUgKHRoZSBsb2cgZm9sZCBjaGFuZ2UgbWF5IHNvbWV0aW1lcyBiZSByZWZlcnJlZCB0byBhcyBhbiBlZmZlY3Qgc2l6ZSksIHRoZSBgREVTZXEyYCBhdXRob3JzIHJlY29tbWVuZCAic2hyaW5raW5nIiB0aGUgbG9nIGZvbGQtY2hhbmdlcyB3aGljaCBpcyBhdmFpbGFibGUgaW4gREVTZXEy4oCZcyBgbGZjU2hyaW5rYCBmdW5jdGlvbi4gVGhpcyByZXN1bHRzIGluIG1vcmUgc3RhYmxlIGZvbGQgY2hhbmdlIHZhbHVlcy4gVGhlIHAtdmFsdWVzIGFyZSB1bmFmZmVjdGVkLgoKYGBge3J9CnJlc19MdnNWIDwtIGxmY1NocmluayhkZS5tZixjb250cmFzdD1jKCJTdGF0dXMiLCJsYWN0YXRpb24iLCJ2aXJnaW4iKSkKREVTZXEyOjpwbG90TUEocmVzX0x2c1YpCmBgYAoKV2Ugd2lsbCByZS1kZWZpbmUgb3VyIHJlc3VsdHMgb2JqZWN0IHRvIHVzZSB0aGVzZSBuZXcgZm9sZC1jaGFuZ2VzLgoKYGBge3J9CnJlc3VsdHMub3JkZXJlZCA8LSBhcy5kYXRhLmZyYW1lKHJlc19MdnNWKSAlPiUgCiAgcm93bmFtZXNfdG9fY29sdW1uKCJFTlNFTUJMIikgJT4lIAogIGFycmFuZ2UocGFkaikKaGVhZChyZXN1bHRzLm9yZGVyZWQpCmBgYAoKQW5vdGhlciBjb21tb24gcGxvdCBmb3IgZGlzcGxheWluZyB0aGUgcmVzdWx0cyBvZiBhIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIGlzIGEgKnZvbGNhbm8gcGxvdCoKCmBgYHtyfQpsaWJyYXJ5KGdncGxvdDIpCnJlc3VsdHMub3JkZXJlZCAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gbG9nMkZvbGRDaGFuZ2UsIHkgPSAtbG9nMTAocGFkaikpKSArIGdlb21fcG9pbnQoKQoKYGBgCgoKSXQgY2FuIGFsc28gYmUgdXNlZnVsIHRvIGV4YW1pbmUgdGhlIGNvdW50cyBvZiByZWFkcyBmb3IgYSBzaW5nbGUgZ2VuZSBhY3Jvc3MgdGhlIGdyb3Vwcy4gQSBzaW1wbGUgZnVuY3Rpb24gZm9yIG1ha2luZyB0aGlzIHBsb3QgaXMgYHBsb3RDb3VudHNgLCB3aGljaCBub3JtYWxpemVzIGNvdW50cyBieSBzZXF1ZW5jaW5nIGRlcHRoIGFuZCBhZGRzIGEgcHNldWRvY291bnQgb2YgMS8yIHRvIGFsbG93IGZvciBsb2cgc2NhbGUgcGxvdHRpbmcuIFRoZSBjb3VudHMgYXJlIGdyb3VwZWQgYnkgdGhlIHZhcmlhYmxlcyBpbiAgYGludGdyb3VwYCwgd2hlcmUgbW9yZSB0aGFuIG9uZSB2YXJpYWJsZSBjYW4gYmUgc3BlY2lmaWVkLiBIZXJlIHdlIHNwZWNpZnkgdGhlIGdlbmUgd2hpY2ggaGFkIHRoZSBzbWFsbGVzdCBwIHZhbHVlIGZyb20gdGhlIHJlc3VsdHMgdGFibGUgY3JlYXRlZCBhYm92ZS4gWW91IGNhbiBzZWxlY3QgdGhlIGdlbmUgdG8gcGxvdCBieSByb3duYW1lIG9yIGJ5IG51bWVyaWMgaW5kZXg6LQoKYGBge3J9CnBsb3RDb3VudHMoZGRzLCAiRU5TTVVTRzAwMDAwMDAwMzgxIixpbnRncm91cCA9IGMoIlN0YXR1cyIpKQpgYGAKCklmIHdlIHdhbnQgZ3JlYXRlciBjb250cm9sIG92ZXIgaG93IHRvIHZpc3VhbGlzZSB0aGUgZGF0YSwgd2UgY2FuIHVzZSB0aGUgYHBsb3RDb3VudHNgIGZ1bmN0aW9uIHRvIHJldHVybiB0aGUgY291bnQgZGF0YSwgYnV0IG5vdCBhY3R1YWxseSBwcm9kdWNlIHRoZSBwbG90Oi0KCmBgYHtyfQpwbG90Q291bnRzKGRkcywgIkVOU01VU0cwMDAwMDAwMDM4MSIsaW50Z3JvdXAgPSBjKCJTdGF0dXMiKSxyZXR1cm5EYXRhPVRSVUUpCmBgYAoKCj4gIyMgQ2hhbGxlbmdlIDEgey5jaGFsbGVuZ2V9Cj4KPiAxLiBVc2UgdGhlIG9wdGlvbiBgcmV0dXJuRGF0YT1UUlVFYCB0byBnZXQgYSBkYXRhIGZyYW1lIGNvbnRhaW5pbmcgdGhlIGNvdW50cyBvZiBgRU5TTVVTRzAwMDAwMDAwMzgxYCBpbiB0aGUgZGlmZmVyZW50IGRldmVsb3BtZW50IHN0YWdlcy4gVmlzdWFsaXNlIHRoZXNlIGRhdGEgdXNpbmcgYGdncGxvdDJgIChzZWUgcGxvdCBBIGJlbG93KS4gCj4gMi4gUmVwZWF0IHRoZSB2b2xjYW5vIHBsb3QgZnJvbSBhYm92ZSwgYnV0IHVzZSBhIGRpZmZlcmVudCBjb2xvdXIgdG8gaW5kaWNhdGUgd2hpY2ggZ2VuZXMgYXJlIHNpZ25pZmljYW50IHdpdGggYW4gYWRqdXN0ZWQgcC12YWx1ZSBsZXNzIHRoYW4gMC4wNS4gU2VlIHBsb3QgQiBiZWxvdwo+IDMuIChPcHRpb25hbCkgVGhlIGFyZ3VtZW50IGBpbnRncm91cD1gIGNhbiBiZSB1c2VkIHRvIHJldHJpZXZlIGFuZCBwbG90IGRhdGEgZnJvbSBtdWx0aXBsZSB2YXJpYWJsZXMgb2YgaW50ZXJlc3QgaW4gdGhlIGRhdGEuIFVzZSB0aGUgdmFsdWUgYGludGdyb3VwPWMoIlN0YXR1cyIsIkNlbGxUeXBlIilgIGFuZCBjb21wYXJlIHRoZSBjb3VudHMgYmV0d2VlbiBkaWZmZXJlbnQgY2VsbCB0eXBlcyBhbmQgc3RhdHVzLiBTZWUgcGxvdCBDIGJlbG93Lgo+IEhJTlQ6IFRvIGdldCB0aGUgY291bnRzIG9uIHRoZSBzYW1lIHNjYWxlIGFzIGRpc3BsYXllZCBieSB0aGUgcGxvdENvdW50cyBmdW5jdGlvbiB5b3Ugd2lsbCBuZWVkIHRvIGFkZCBgK3NjYWxlX3lfbG9nMTBgIGluIHlvdXIgZ2dwbG90MiBjb2RlCgpgYGB7ciBlY2hvPUZBTFNFfQpwMSA8LSBwbG90Q291bnRzKGRkcywgIkVOU01VU0cwMDAwMDAwMDM4MSIsaW50Z3JvdXAgPSBjKCJTdGF0dXMiLCJDZWxsVHlwZSIpLHJldHVybkRhdGEgPSBUUlVFKSAlPiUgICBnZ3Bsb3QoYWVzKHggPSBTdGF0dXMsIHkgPSBjb3VudCxjb2w9U3RhdHVzKSkgKyBnZW9tX2ppdHRlcih3aWR0aD0wLjEpICsgc2NhbGVfeV9sb2cxMCgpCnAyIDwtIHJlc3VsdHMub3JkZXJlZCAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gbG9nMkZvbGRDaGFuZ2UsIHkgPSAtbG9nMTAocGFkaiksIGNvbD1wYWRqIDwgMC4wNSkpICsgZ2VvbV9wb2ludCgpCnAzIDwtIHBsb3RDb3VudHMoZGRzLCAiRU5TTVVTRzAwMDAwMDAwMzgxIixpbnRncm91cCA9IGMoIlN0YXR1cyIsIkNlbGxUeXBlIikscmV0dXJuRGF0YSA9IFRSVUUpICU+JSAgIGdncGxvdChhZXMoeCA9IFN0YXR1cywgeSA9IGNvdW50LGNvbD1TdGF0dXMpKSArIGdlb21faml0dGVyKHdpZHRoPTAuMSkgKyBmYWNldF93cmFwKH5DZWxsVHlwZSkgKyBzY2FsZV95X2xvZzEwKCkKCmNvd3Bsb3Q6OnBsb3RfZ3JpZChwMSxwMixwMyxsYWJlbHM9TEVUVEVSU1sxOjNdKQoKYGBgCgoKSG93ZXZlciwgaXQgaXMgaGFyZCB0byBhc3Nlc3MgdGhlIGJpb2xvZ2ljYWwgc2lnbmlmaWNhbmNlIG9mIHN1Y2ggYSBnZW5lIHdpdGhvdXQgbW9yZSBpbmZvcm1hdGlvbiBhYm91dCAuIFRvIHBlcmZvcm0gc3VjaCBhIHRhc2sgd2UgbmVlZCB0byBtYXAgYmV0d2VlbiB0aGUgaWRlbnRpZmllcnMgd2UgaGF2ZSBpbiB0aGUgYERFU2VxMmAgb3V0cHV0IGFuZCBtb3JlIGZhbWlsaWFyIG5hbWVzLgoKCiMjIEFkZGluZyBhbm5vdGF0aW9uIHRvIHRoZSBERVNlcTIgcmVzdWx0cwoKVGhlcmUgYXJlIGEgbnVtYmVyIG9mIHdheXMgdG8gYWRkIGFubm90YXRpb24sIGJ1dCB3ZSB3aWxsIGRlbW9uc3RyYXRlIGhvdyB0byBkbyB0aGlzIHVzaW5nIHRoZSAqb3JnLk1tLmVnLmRiKiBwYWNrYWdlLiBUaGlzIHBhY2thZ2UgaXMgb25lIG9mIHNldmVyYWwgKm9yZ2FuaXNtLWxldmVsKiBwYWNrYWdlcyB3aGljaCBhcmUgcmUtYnVpbHQgZXZlcnkgNiBtb250aHMuIFRoZXNlIHBhY2thZ2VzIGFyZSBsaXN0ZWQgb24gdGhlIFthbm5vdGF0aW9uIHNlY3Rpb25dKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL3JlbGVhc2UvQmlvY1ZpZXdzLmh0bWwjX19fQW5ub3RhdGlvbkRhdGEpIG9mIHRoZSBCaW9jb25kdWN0b3IsIGFuZCBhcmUgaW5zdGFsbGVkIGluIHRoZSBzYW1lIHdheSBhcyByZWd1bGFyIEJpb2NvbmR1Y3RvciBwYWNrYWdlcy4gQW4gYWx0ZXJuYXRpdmUgYXBwcm9hY2ggaXMgdG8gdXNlIGBiaW9tYVJ0YCwgYW4gaW50ZXJmYWNlIHRvIHRoZSBbQmlvTWFydF0oaHR0cDovL3d3dy5iaW9tYXJ0Lm9yZy8pIHJlc291cmNlLiBCaW9NYXJ0IGlzIG11Y2ggbW9yZSBjb21wcmVoZW5zaXZlLCBidXQgdGhlIG9yZ2FuaXNtIHBhY2thZ2VzIGZpdCBiZXR0ZXIgaW50byB0aGUgQmlvY29uZHVjdG9yIHdvcmtmbG93LgoKCmBgYHtyIGV2YWw9RkFMU0V9CiMjIyBPbmx5IGV4ZWN1dGUgd2hlbiB5b3UgbmVlZCB0byBpbnN0YWxsIHRoZSBwYWNrYWdlCmluc3RhbGwucGFja2FnZXMoIkJpb2NNYW5hZ2VyIikKQmlvY01hbmFnZXI6Omluc3RhbGwoIm9yZy5NbS5lZy5kYiIpCiMgRm9yIEh1bWFuCkJpb2NNYW5hZ2VyOjppbnN0YWxsKCJvcmcuSHMuZWcuZGIiKQpgYGAKClRoZSBwYWNrYWdlcyBhcmUgbGFyZ2VyIGluIHNpemUgdGhhdCBCaW9jb25kdWN0b3Igc29mdHdhcmUgcGFjYWtnZXMsIGJ1dCBlc3NlbnRpYWxseSB0aGV5IGFyZSBkYXRhYmFzZXMgdGhhdCBjYW4gYmUgdXNlZCB0byBtYWtlICpvZmZsaW5lKiBxdWVyaWVzLiAKCmBgYHtyIG1lc3NhZ2U9RkFMU0V9CmxpYnJhcnkob3JnLk1tLmVnLmRiKQpgYGAKCgpGaXJzdCB3ZSBuZWVkIHRvIGRlY2lkZSB3aGF0IGluZm9ybWF0aW9uIHdlIHdhbnQuIEluIG9yZGVyIHRvIHNlZSB3aGF0IHdlIGNhbiBleHRyYWN0IHdlIGNhbiBydW4gdGhlIGBjb2x1bW5zYCBmdW5jdGlvbiBvbiB0aGUgYW5ub3RhdGlvbiBkYXRhYmFzZS4KCmBgYHtyfQpjb2x1bW5zKG9yZy5NbS5lZy5kYikKYGBgCgpXZSBhcmUgZ29pbmcgdG8gZmlsdGVyIHRoZSBkYXRhYmFzZSBieSBhIGtleSBvciBzZXQgb2Yga2V5cyBpbiBvcmRlciB0byBleHRyYWN0IHRoZSBpbmZvcm1hdGlvbiB3ZSB3YW50LiBWYWxpZCBuYW1lcyBmb3IgdGhlIGtleSBjYW4gYmUgcmV0cmlldmVkIHdpdGggdGhlIGBrZXl0eXBlc2AgZnVuY3Rpb24uCgpgYGB7cn0Ka2V5dHlwZXMob3JnLk1tLmVnLmRiKQpgYGAKCldlIHNob3VsZCBzZWUgYEVOU0VNQkxgLCB3aGljaCBpcyB0aGUgdHlwZSBvZiBrZXkgd2UgYXJlIGdvaW5nIHRvIHVzZSBpbiB0aGlzIGNhc2UuIElmIHdlIGFyZSB1bnN1cmUgd2hhdCB2YWx1ZXMgYXJlIGFjY2VwdGFibGUgZm9yIHRoZSBrZXksIHdlIGNhbiBjaGVjayB3aGF0IGtleXMgYXJlIHZhbGlkIHdpdGggYGtleXNgCgpgYGB7cn0Ka2V5cyhvcmcuTW0uZWcuZGIsIGtleXR5cGU9IkVOU0VNQkwiKVsxOjEwXQpgYGAKCgoKRm9yIHRoZSB0b3AgZ2VuZSBpbiBvdXIgYW5hbHlzaXMgdGhlIGNhbGwgdG8gdGhlIGZ1bmN0aW9uIHdvdWxkIGJlOi0KCmBgYHtyIGV2YWw9RkFMU0V9CnNlbGVjdChvcmcuTW0uZWcuZGIsIGtleXM9IkVOU01VU0cwMDAwMDAwMDM4MSIsCiAgICAgICBrZXl0eXBlID0gIkVOU0VNQkwiLGNvbHVtbnM9YygiU1lNQk9MIiwiR0VORU5BTUUiKQopCgpgYGAKClVuZm9ydHVuYXRlbHksIHRoZSBhdXRob3JzIG9mIGBkcGx5cmAgYW5kIGBBbm5vdGF0aW9uRGJpYCBoYXZlIGJvdGggZGVjaWRlZCB0byB1c2UgdGhlIG5hbWUgYHNlbGVjdGAgaW4gdGhlaXIgcGFja2FnZXMuIFRvIGF2b2lkIGNvbmZ1c2lvbiwgdGhlIGZvbGxvd2luZyBjb2RlIGlzIHNvbWV0aW1lcyB1c2VkOi0KCmBgYHtyfQpBbm5vdGF0aW9uRGJpOjpzZWxlY3Qob3JnLk1tLmVnLmRiLCBrZXlzPSJFTlNNVVNHMDAwMDAwMDAzODEiLGtleXR5cGUgPSAiRU5TRU1CTCIsY29sdW1ucz1jKCJTWU1CT0wiLCJHRU5FTkFNRSIpKQpgYGAKCgpUbyBhbm5vdGF0ZSBvdXIgcmVzdWx0cywgd2UgZGVmaW5pdGVseSB3YW50IGdlbmUgc3ltYm9scyBhbmQgcGVyaGFwcyB0aGUgZnVsbCBnZW5lIG5hbWUuIExldCdzIGJ1aWxkIHVwIG91ciBhbm5vdGF0aW9uIGluZm9ybWF0aW9uIGludG8gYSBuZXcgZGF0YSBmcmFtZSB1c2luZyB0aGUgYHNlbGVjdGAgZnVuY3Rpb24uCgpgYGB7cn0KYW5ubyA8LSBBbm5vdGF0aW9uRGJpOjpzZWxlY3Qob3JnLk1tLmVnLmRiLGtleXM9cmVzdWx0cy5vcmRlcmVkJEVOU0VNQkwsCiAgICAgICAgICAgICAgY29sdW1ucz1jKCJTWU1CT0wiLCJHRU5FTkFNRSIpLAogICAgICAgICAgICAgIGtleXR5cGU9IkVOU0VNQkwiKQojIEhhdmUgYSBsb29rIGF0IHRoZSBhbm5vdGF0aW9uCmhlYWQoYW5ubykKCmBgYAoKSG93ZXZlciwgd2UgaGF2ZSBhIHByb2JsZW0gdGhhdCB0aGUgcmVzdWx0aW5nIGRhdGEgZnJhbWUgaGFzIG1vcmUgcm93cyB0aGFuIG91ciByZXN1bHRzIHRhYmxlLiBUaGlzIGlzIGR1ZSB0byB0aGUgKm9uZS10by1tYW55KiByZWxhdGlvbnNoaXBzIHRoYXQgb2Z0ZW4gb2NjdXIgd2hlbiBtYXBwaW5nIGJldHdlZW4gdmFyaW91cyBpZGVudGlmaWVycy4KCmBgYHtyfQpkaW0oYW5ubykKZGltKHJlc3VsdHMub3JkZXJlZCkKYGBgCgpTdWNoIGR1cGxpY2F0ZWQgZW50cmllcyBjYW4gYmUgaWRlbnRpZmllZCB1c2luZyB0aGUgYGR1cGxpY2F0ZWRgIGZ1bmN0aW9uLiAKCmBgYHtyfQpkdXBfaWRzIDwtIGFubm8kRU5TRU1CTFtkdXBsaWNhdGVkKGFubm8kRU5TRU1CTCldCmZpbHRlcihhbm5vLCBFTlNFTUJMICVpbiUgZHVwX2lkcykgJT4lIAogIGFycmFuZ2UoRU5TRU1CTCkgJT4lIGhlYWQKCmBgYAoKRm9ydHVuYXRlbHksIHRoZXJlIGFyZSBub3QgdG9vIG1hbnkgc28gaG9wZWZ1bGx5IHdlIHdvbid0IGxvc2UgdG9vIG11Y2ggaW5mb3JtYXRpb24gaWYgd2UgZGlzY2FyZCB0aGUgZW50cmllcyB0aGF0IGFyZSBkdXBsaWNhdGVkLiBUaGUgZmlyc3Qgb2NjdXJlbmNlIG9mIHRoZSBkdXBsaWNhdGVkIElEIHdpbGwgc3RpbGwgYmUgaW5jbHVkZWQgaW4gdGhlIHRhYmxlLgoKYGBge3J9CmFubm8gPC0gQW5ub3RhdGlvbkRiaTo6c2VsZWN0KG9yZy5NbS5lZy5kYixrZXlzPXJlc3VsdHMub3JkZXJlZCRFTlNFTUJMLAogICAgICAgICAgICAgIGNvbHVtbnM9YygiRU5TRU1CTCIsIlNZTUJPTCIsIkdFTkVOQU1FIiwiRU5UUkVaSUQiKSwKICAgICAgICAgICAgICBrZXl0eXBlPSJFTlNFTUJMIikgJT4lIAogIGZpbHRlcighZHVwbGljYXRlZChFTlNFTUJMKSkKZGltKGFubm8pCmBgYAoKCldlIGNhbiBiaW5kIGluIHRoZSBhbm5vdGF0aW9uIGluZm9ybWF0aW9uIHRvIHRoZSBgcmVzdWx0c2AgZGF0YSBmcmFtZS4gCgpgYGB7cn0KcmVzdWx0cy5hbm5vdGF0ZWQgPC0gbGVmdF9qb2luKHJlc3VsdHMub3JkZXJlZCwgYW5ubyxieT0iRU5TRU1CTCIpCmhlYWQocmVzdWx0cy5hbm5vdGF0ZWQpCgpgYGAKCgpXZSBjYW4gc2F2ZSB0aGUgcmVzdWx0cyB0YWJsZSB1c2luZyB0aGUgYHdyaXRlLmNzdmAgZnVuY3Rpb24sIHdoaWNoIHdyaXRlcyB0aGUgcmVzdWx0cyBvdXQgdG8gYSBjc3YgZmlsZSB0aGF0IHlvdSBjYW4gb3BlbiBpbiBleGNlbC4KCmBgYHtyfQp3cml0ZS5jc3YocmVzdWx0cy5hbm5vdGF0ZWQsZmlsZT0idmlyZ2luX3ZzX2xhY3RhdGlvbl9ERVNlcV9hbm5vdGF0ZWQuY3N2Iixyb3cubmFtZXM9RkFMU0UpCmBgYAoKCgoKV2UgaGF2ZSBhbHJlYWR5IHNlZW4gdGhlIHVzZSBvZiBhIGhlYXRtYXAgYXMgYSBxdWFsaXR5IGFzc2Vzc21lbnQgdG9vbCB0byB2aXN1YWxpc2UgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHNhbXBsZXMgaW4gYW4gZXhwZXJpbWVudC4gQW5vdGhlciBjb21tb24gdXNlLWNhc2UgZm9yIHN1Y2ggYSBwbG90IGlzIHRvIHZpc3VhbGlzZSB0aGUgcmVzdWx0cyBvZiBhIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzLgoKSGVyZSB3ZSB3aWxsIHRha2UgdGhlIHRvcCAxMCBnZW5lcyBmcm9tIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcyBhbmQgcHJvZHVjZSBhIGhlYXRtYXAgd2l0aCB0aGUgYHBoZWF0bWFwYCBwYWNrYWdlLiBUaGUgZGVmYXVsdCBjb2xvdXIgcGFsZXR0ZSBnb2VzIGZyb20gbG93IGV4cHJlc3Npb24gaW4gYmx1ZSB0byBoaWdoIGV4cHJlc3Npb24gaW4gcmVkLCB3aGljaCBpcyBhIGdvb2QgYWx0ZXJuYXRpdmUgdG8gdGhlIHRyYWRpdGlvbmFsIHJlZC9ncmVlbiBoZWF0bWFwcyB3aGljaCBhcmUgbm90IHN1aXRhYmxlIGZvciB0aG9zZSB3aXRoIGZvcm1zIG9mIGNvbG91ci1ibGluZG5lc3MuCgpUaGUgY291bnRzIHdlIGFyZSB2aXN1YWxpc2luZyBhcmUgdGhlICp2YXJpYW5jZS1zdGFibGlzZWQqIGNvdW50cywgd2hpY2ggYXJlIG1vcmUgYXBwcm9wcmlhdGUgZm9yIHZpc3VhbGlzYXRpb24uCgpgYGB7cn0KbGlicmFyeShwaGVhdG1hcCkKdG9wX2dlbmVzIDwtIHJlc3VsdHMuYW5ub3RhdGVkJEVOU0VNQkxbMToxMF0KCnZzZCA8LSB2c3QoZGRzKQpwaGVhdG1hcChhc3NheSh2c2QpW3RvcF9nZW5lcyxdKQoKCmBgYAoKVGhlIGhlYXRtYXAgaXMgbW9yZSBpbmZvcm1hdGl2ZSBpZiB3ZSBhZGQgY29sb3VycyB1bmRlcm5lYXRoIHRoZSBzYW1wbGUgZGVuZHJvZ3JhbSB0byBpbmRpY2F0ZSB3aGljaCBzYW1wbGUgZ3JvdXAgZWFjaCBzYW1wbGUgYmVsb25ncyB0by4gVGhpcyB3ZSBjYW4gZG8gYnkgY3JlYXRpbmcgYSBkYXRhIGZyYW1lIGNvbnRhaW5pbmcgbWV0YWRhdGEgZm9yIGVhY2ggb2YgdGhlIHNhbXBsZXMgaW4gb3VyIGRhdGFzZXQuIFdpdGggdGhlIGBERVNlcTJgIHdvcmtmbG93IHdlIGhhdmUgYWxyZWFkeSBjcmVhdGVkIHN1Y2ggYSBkYXRhIGZyYW1lLiBXZSBoYXZlIHRvIG1ha2Ugc3VyZSB0aGUgdGhlIHJvd25hbWVzIG9mIHRoZSBkYXRhIGZyYW1lIGFyZSB0aGUgc2FtZSBhcyB0aGUgY29sdW1uIG5hbWVzIG9mIHRoZSBjb3VudHMgbWF0cml4LgoKYGBge3J9CnNhbXBsZUluZm8gPC0gYXMuZGF0YS5mcmFtZShjb2xEYXRhKGRkcylbLGMoIlN0YXR1cyIsIkNlbGxUeXBlIildKQoKcGhlYXRtYXAoYXNzYXkodnNkKVt0b3BfZ2VuZXMsXSwKICAgICAgICAgYW5ub3RhdGlvbl9jb2wgPSBzYW1wbGVJbmZvKQpgYGAKCkFueSBwbG90IHdlIGNyZWF0ZSBpbiBSU3R1ZGlvIGNhbiBiZSBzYXZlZCBhcyBhIHBuZyBvciBwZGYgZmlsZS4gV2UgdXNlIHRoZSBgcG5nYCBvciBgcGRmYCBmdW5jdGlvbiB0byBjcmVhdGUgYSBmaWxlIGZvciB0aGUgcGxvdCB0byBiZSBzYXZlZCBpbnRvIGFuZCBydW4gdGhlIHJlc3Qgb2YgdGhlIGNvZGUgYXMgbm9ybWFsLiBUaGUgcGxvdCBkb2VzIG5vdCBnZXQgZGlzcGxheWVkIGluIFJTdHVkaW8sIGJ1dCBwcmludGVkIHRvIHRoZSBzcGVjaWZpZWQgZmlsZS4gCgpgYGB7cn0KCnBuZygiaGVhdG1hcF90b3AxMF9nZW5lcy5wbmciLHdpZHRoPTgwMCxoZWlnaHQ9ODAwKQpwaGVhdG1hcChhc3NheSh2c2QpW3RvcF9nZW5lcyxdLAogICAgICAgICBhbm5vdGF0aW9uX2NvbCA9IHNhbXBsZUluZm8pCiMgZGV2Lm9mZigpCmBgYAoKCj4gIyMgQ2hhbGxlbmdlIDJ7LmNoYWxsZW5nZX0KPiAxLiBSZXBlYXQgdGhlIHNhbWUgaGVhdG1hcCBhcyBhYm92ZSwgYnV0IGZvciB0aGUgdG9wIDEwMCBtb3N0IGRpZmZlcmVudGlhbGx5LWV4cHJlc3NlZCBnZW5lcyAqKmJldHdlZW4gcHJlZ25hbnQgYW5kIGx1bWluYWwqKgo+IDIuIENoYW5nZSB0aGUgcGxvdCBzbyB0aGF0IGdlbmUgbmFtZXMgYXJlIGRpc3BsYXllZCByYXRoZXIgdGhhbiBFbnNlbWJsIElEcwo+IDMuIFNhdmUgdGhlIHBsb3QgdG8gYSBwZGYgZmlsZQo+IEhJTlQ6IGNoZWNrIHRoZSBoZWxwIGZvciB0aGUgYHBoZWF0bWFwYCBmdW5jdGlvbiB0byBzZWUgaG93IGNvbHVtbiBhbmQgcm93IGxhYmVscyBjYW4gYmUgY2hhbmdlZAoKIyMjIEFjY2Vzc2luZyB0aGUgc2FtcGxlIG9yIGdlbmUgY2x1c3RlcnMKClRoZSBoZWF0bWFwIGRpc3BsYXlzIHJlbGF0aW9uc2hpcHMgYmV0d2VlbiBzYW1wbGVzIGFuZCBnZW5lcyBpbiBvdXIgc3R1ZHkgYXMgYSB1c2VmdWwgdmlzdWFsaXNhdGlvbi4gSW4gdGhpcyBleGFtcGxlIHdlIGNhbiBlYXNpbHkgaWRlbnRpZnkgd2hpY2ggc2FtcGxlcyBhcmUgbW9zdCBzaW1pbGFyIGJhc2VkIG9uIHRoZWlyIGV4cHJlc3Npb24gcGF0dGVybnMuIEhvd2V2ZXIsIGZvciBsYXJnZXIgZGF0YXNldCB0aGlzIG1heSBiZSBtb3JlIHByb2JsZW1hdGljLiBXZSBjYW4gZXh0cmFjdCBkYXRhIHRoZSBzYW1wbGUgcmVsYXRpb25zaGlwcyBhYm91dCBpZiB3ZSBtYW51YWxseSBwZXJmb3JtIHRoZSBjbHVzdGVyaW5nIHN0ZXBzIHVzZWQgYnkgYHBoZWF0bWFwYC4gRmlyc3QgaXMgdG8gY2x1c3RlciB0aGUgc2FtcGxlcyB3aXRoIHRoZSBkZWZhdWx0IGRpc3RhbmNlIG1hdHJpeCBhbmQgY2x1c3RlcmluZyBhbGdvcml0aG1zLiAKCmBgYHtyfQptYXQgPC0gYXNzYXkodnNkKVt0b3BfZ2VuZXMsXQojIyBDYWxjdWxhdGUgdGhlIGRpc3RhbmNlIG1hdHJpeCBiZXR3ZWVuIHNhbXBsZXMKZF9zYW1wbGVzIDwtIGRpc3QodChtYXQpKQoKcGxvdChoY2x1c3QoZF9zYW1wbGVzKSkKcmVjdC5oY2x1c3QoaGNsdXN0KGRfc2FtcGxlcyksaz0yKQpgYGAKCldlIGNhbiB0aGVuICJjdXQiIHRoZSBkZW5kcm9ncmFtIHRvIGdpdmUgYSBzZXQgbnVtYmVyIG9mIGNsdXN0ZXJzLiBFYWNoIHNhbXBsZSBoYXMgYmVlbiBhc3NpZ25lZCBhIGxhYmVsIG9mIGAxYCBvciBgMmAgZGVwZW5kaW5nIG9uIHdoaWNoIGNsdXN0ZXIgaXQgYmVsb25ncyB0by4KCmBgYHtyfQpjbHVzdGVycyA8LSBjdXRyZWUoaGNsdXN0KGRfc2FtcGxlcyksayA9IDIpCmNsdXN0ZXJzCgpgYGAKClRoZSBncm91cGluZ3MgY291bGQgdGhlbiBiZSB0YWJ1bGF0ZWQgYWdhaW5zdCB3aXRoIHRoZSBzYW1wbGUgbWV0YWRhdGEgdG8gc2VlIGlmIHBhcnRpY3VsYXIgYmlvbG9naWNhbCBncm91cHMgYXJlIGFzc29jaWF0ZWQgd2l0aCB0aGUgbmV3IGNsdXN0ZXJzIHdlIGhhdmUgaWRlbnRpZmllZC4KCmBgYHtyfQp0YWJsZShjbHVzdGVycywgY29sRGF0YShkZHMpJFN0YXR1cykKCmBgYAoKIyMjIEFkZGluZyBnZW5lIG5hbWVzIHRvIGEgdm9sY2FubyBwbG90CgpOb3cgdGhhdCB3ZSBoYXZlIGFuIGFubm90YXRlZCB0YWJsZSBvZiByZXN1bHRzLCB3ZSBjYW4gYWRkIHRoZSBnZW5lIG5hbWVzIHRvIHNvbWUgb2YgdGhlIG90aGVyIHBsb3RzIHdlIGhhdmUgY3JlYXRlZC4gVGhpcyBzaG91bGQgYmUgc3RyYWlnaHRmb3J3YXJkIGFzIGdncGxvdDIgaGFzIGEgYGxhYmVsYCBhZXN0aGV0aWMgdGhhdCBjYW4gYmUgbWFwcGVkIHRvIGNvbHVtbnMgaW4gYSBkYXRhIGZyYW1lLiBUaGUgYGdlb21fdGV4dGAgcGxvdCB3aWxsIHRoZW4gZGlzcGxheSB0aGUgbGFiZWxzLiBIb3dldmVyLCB0aGUgZm9sbG93aW5nIHBsb3QgaXMgYSBiaXQgY3Jvd2RlZC4KCmBgYHtyfQojIyBOb3QgYSBnb29kIGlkZWEgdG8gcnVuIHRoaXMhIQpyZXN1bHRzLmFubm90YXRlZCAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gbG9nMkZvbGRDaGFuZ2UsIHkgPSAtbG9nMTAocGFkaiksIGxhYmVsPVNZTUJPTCkpICsgZ2VvbV9wb2ludCgpICsgZ2VvbV90ZXh0KCkKYGBgCgoKVGhlIHByb2JsZW0gaGVyZSBpcyB0aGF0IGdncGxvdDIgaXMgdHJ5aW5nIHRvIGxhYmVsIGV2ZXJ5IHBvaW50IHdpdGggYSBuYW1lOyBub3QgcXVpdGUgd2hhdCB3ZSB3YW50LiBUaGUgdHJpY2sgaXMgdG8gY3JlYXRlIGEgbGFiZWwgdGhhdCBpcyBibGFuayBmb3IgbW9zdCBnZW5lcyBhbmQgb25seSBsYWJlbHMgdGhlIHBvaW50cyB3ZSBhcmUgaW50ZXJlc3RlZCBpbi4gVGhlIGBpZmVsc2VgIGZ1bmN0aW9uIGluIFIgaXMgYSBjb252ZW5pZW50IHdheSB0byBzZXQgdGhlIGVudHJpZXMgaW4gYSB2ZWN0b3IgYmFzZWQgb24gYSAqbG9naWNhbCogZXhwcmVzc2lvbi4gSW4gdGhpcyBjYXNlLCBtYWtlIHRoZSB2YWx1ZXMgaW4gYExhYmVsYCB0aGUgc2FtZSBhcyB0aGUgZ2VuZSBzeW1ib2wgaWYgdGhlIGdlbmUgaXMgaW4gb3VyIGxpc3Qgb2YgInRvcCBnZW5lcyIuIE90aGVyd2lzZSwgcG9pbnRzIGdldCBsYWJlbGVkIHdpdGggYSBibGFuayBzdHJpbmcgYCIiYC4KCkZvciBjbGFyaXR5LCB3ZSBhbHNvIG1ha2UgdGhlIHBvaW50cyBzbGlnaHRseSB0cmFuc3BhcmVudCBhbmQgdXNlIGEgZGlmZmVyZW50IGNvbG91ciBmb3IgdGhlIHRleHQuCgpgYGB7cn0KTiA8LSAxMAp0b3BfZ2VuZXMgPC0gcmVzdWx0cy5hbm5vdGF0ZWQkRU5TRU1CTFsxOk5dCnJlc3VsdHMuYW5ub3RhdGVkICU+JSAKICBtdXRhdGUoTGFiZWwgPSBpZmVsc2UoRU5TRU1CTCAlaW4lIHRvcF9nZW5lcywgU1lNQk9MLCAiIikpICU+JSAgCiAgZ2dwbG90KGFlcyh4ID0gbG9nMkZvbGRDaGFuZ2UsIHkgPSAtbG9nMTAocGFkaiksIGxhYmVsPUxhYmVsKSkgKyBnZW9tX3BvaW50KGFscGhhPTAuNCkgKyBnZW9tX3RleHQoY29sPSJibHVlIikKYGBgCgpGaW5hbGx5LCBhIHNsaWdodGx5IGJldHRlciBwb3NpdGlvbmluZyBvZiB0ZXh0IGlzIGdpdmVuIGJ5IHRoZSBgZ2dyZXBlbGAgcGFja2FnZS4KCmBgYHtyfQppZighcmVxdWlyZShnZ3JlcGVsKSkgaW5zdGFsbC5wYWNrYWdlcygiZ2dyZXBlbCIpCgpyZXN1bHRzLmFubm90YXRlZCAlPiUgCiAgbXV0YXRlKExhYmVsID0gaWZlbHNlKEVOU0VNQkwgJWluJSB0b3BfZ2VuZXMsIFNZTUJPTCwgIiIpKSAlPiUgIAogIGdncGxvdChhZXMoeCA9IGxvZzJGb2xkQ2hhbmdlLCB5ID0gLWxvZzEwKHBhZGopLCBsYWJlbD1MYWJlbCkpICsgZ2VvbV9wb2ludChhbHBoYT0wLjQpICsgZ2VvbV90ZXh0X3JlcGVsKGNvbD0iYmx1ZSIpCmBgYAoKCiMjIyBBbm5vdGF0aW9uIHdpdGggdGhlIGJpb21hUnQgcmVzb3VyY2UKClRoZSBCaW9jb25kdWN0b3IgcGFja2FnZSBoYXZlIHRoZSBjb252ZW5pZW5jZSBvZiBiZWluZyBhYmxlIHRvIG1ha2UgcXVlcmllcyBvZmZsaW5lLiBIb3dldmVyLCB0aGV5IGFyZSBvbmx5IGF2YWlsYWJsZSBmb3IgY2VydGFpbiBvcmdhbmlzbXMuIElmIHlvdXIgb3JnYW5pc20gZG9lcyBub3QgaGF2ZSBhbiBgb3JnLlhYLmVnLmRiYCBwYWNrYWdlIGxpc3RlZCBvbiB0aGUgQmlvY29uZHVjdG9yIGFubm90YXRpb24gcGFnZSAoaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9CaW9jVmlld3MuaHRtbCNfX19Bbm5vdGF0aW9uRGF0YSksIGFuIGFsdGVybmF0aXZlIGlzIHRvIHVzZSBiaW9tYVJ0IHdoaWNoIHByb3ZpZGVzIGFuIGludGVyZmFjZSB0byB0aGUgcG9wdWxhciBiaW9tYXJ0IGFubm90YXRpb24gcmVzb3VyY2UuIAoKVGhlIGZpcnN0IHN0ZXAgaXMgdG8gZmluZCB0aGUgbmFtZSBvZiBhIGRhdGFiYXNlIHRoYXQgeW91IHdhbnQgdG8gY29ubmVjdCB0bwoKYGBge3J9CmxpYnJhcnkoYmlvbWFSdCkKbGlzdE1hcnRzKCkKZW5zZW1ibD11c2VNYXJ0KCJFTlNFTUJMX01BUlRfRU5TRU1CTCIpCiMgbGlzdCB0aGUgYXZhaWxhYmxlIGRhdGFzZXRzIChzcGVjaWVzKS4gUmVwbGFjZSBtb3VzZSB3aXRoIHRoZSBuYW1lIG9mIHlvdXIgb3JnYW5pc20KbGlzdERhdGFzZXRzKGVuc2VtYmwpICU+JSBmaWx0ZXIoZ3JlcGwoIk1vdXNlIixkZXNjcmlwdGlvbikpCgpgYGAKCmBgYHtyfQplbnNlbWJsID0gdXNlRGF0YXNldCgibW11c2N1bHVzX2dlbmVfZW5zZW1ibCIsIG1hcnQ9ZW5zZW1ibCkKYGBgCgpRdWVyaWVzIHRvIGBiaW9tYVJ0YCBhcmUgY29uc3RydWN0ZWQgaW4gYSBzaW1pbGFyIHdheSB0byB0aGUgcXVlcmllcyB3ZSBwZXJmb3JtZWQgd2l0aCB0aGUgYG9yZy5NbS5lZy5kYmAgcGFja2FnZS4gSW5zdGVhZCBvZiBga2V5c2Agd2UgaGF2ZSBgZmlsdGVyc2AsIGFuZCBpbnN0ZWFkIG9mIGBjb2x1bW5zYCB3ZSBoYXZlIGF0dHJpYnV0ZXMuIFRoZSBsaXN0IG9mIGFjY2VwdGFibGUgdmFsdWVzIGlzIG11Y2ggbW9yZSBjb21wcmVoZW5zaXZlIHRoYXQgZm9yIHRoZSBgb3JnLk1tLmVnLmRiYCBwYWNrYWdlLgoKYGBge3J9Cmxpc3RGaWx0ZXJzKGVuc2VtYmwpICU+JSAKICAgIGZpbHRlcihncmVwbCgiZW5zZW1ibCIsbmFtZSkpCmBgYAoKCmBgYHtyIGV2YWw9RkFMU0V9Cmxpc3RBdHRyaWJ1dGVzKGVuc2VtYmwpICU+JSAKICAgIGZpbHRlcihncmVwbCgiZ2VuZSIsbmFtZSkpCmBgYAoKQW4gYWR2YW50YWdlIG92ZXIgdGhlIGBvcmcuLmAgcGFja2FnZXMgaXMgdGhhdCBwb3NpdGlvbmFsIGluZm9ybWF0aW9uIGNhbiBiZSByZXRyaWV2ZWQKCmBgYHtyfQphdHRyaWJ1dGVOYW1lcyA8LSBjKCdlbnNlbWJsX2dlbmVfaWQnLCAnZW50cmV6Z2VuZV9pZCcsICdleHRlcm5hbF9nZW5lX25hbWUnKQoKZ2V0Qk0oYXR0cmlidXRlcyA9IGF0dHJpYnV0ZU5hbWVzLAogICAgICBmaWx0ZXJzID0gImVuc2VtYmxfZ2VuZV9pZCIsCiAgICAgIHZhbHVlcz10b3BfZ2VuZXMsCiAgICAgIG1hcnQ9ZW5zZW1ibCkKYGBgCgo+ICMjIENoYWxsZW5nZSAzey5jaGFsbGVuZ2V9Cj4gMS4gVXNlIGJpb21hUnQgdG8gY3JlYXRlIGFuIGRhdGEgZnJhbWUgY29udGFpbmluZyB0aGUgZW50cmV6Z2VuZSwgZ2VuZSBzeW1ib2wgYW5kIGdlbm9taWMgY29vcmRpbmF0ZXMgKGNocm9tb3NvbWUsIHN0YXJ0LCBlbmQpIGZvciB0aGUgRW5zZW1ibCBJRHMgaW4gdGhlIERFU2VxMiByZXN1bHRzCj4gMi4gUmVtb3ZlIGR1cGxpY2F0ZXMgZW50cmllcyBmcm9tIHRoZSBuZXcgZGF0YSBmcmFtZQo+IDMuIEpvaW4gdGhlIGJpb21hUnQgYW5ub3RhdGlvbiB0byB0aGUgREVTZXEyIHJlc3VsdHMgdG8gcHJvZHVjZSBhIGRhdGEgZnJhbWUgd2l0aCBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiByZXN1bHRzIGFuZCBhbm5vdGF0aW9uCj4gNC4gV3JpdGUgdGhlIGpvaW5lZCBkYXRhIGZyYW1lIHRvIGEgY3N2IGZpbGUKCmBgYHtyfQoKYGBgCgojIyMgT2J0YWluaW5nIGdlbmUgbW9kZWxzIHdpdGggQmlvY29uZHVjdG9yCgpVc2luZyBiaW9tYVJ0IGFzIGFib3ZlIGFsbG93cyB1cyB0byByZXRyaWV2ZSB0aGUgZ2Vub21pYyBjb29yZGluYXRlcyBvZiBhIGdpdmVuIGdlbmUuIElmIHdlIHdhbnQgbW9yZS1jb21wcmVoZW5zaXZlIGluZm9ybWF0aW9uIGFib3V0IHRoZSBzdHJ1Y3R1ZSBvZiBhIGdlbmUgKGFuZCBpbmRlZWQgYWxsIGdlbmVzIGluIHRoZSB0cmFuc2NyaXB0b21lKSwgd2UgY2FuIHVzZSBvbmUgb2Ygc2V2ZXJhbCBwcmUtYnVpbHQgKnRyYW5zY3JpcHQgZGF0YWJhc2UqIHBhY2thZ2VzLgoKUmV0cmlldmluZyB0aGUgY29vcmRpbmF0ZXMgZm9yIGEgcGFydGljdWxhciBnZW5lIChvciBzZXQgb2YgZ2VuZXMpIHVzZXMgdGhlIHNhbWUgYHNlbGVjdGAgZnVuY3Rpb24gYXMgZm9yIHRoZSBgb3JnLk1tLmVnLmRiYCBwYWNrYWdlLCBidXQgY2hlY2tpbmcgZm9yIGRpZmZlcmVudCBgY29sdW1uc2AgYW5kIGBrZXl0eXBlc2AuCgpgYGB7cn0KbGlicmFyeShUeERiLk1tdXNjdWx1cy5VQ1NDLm1tMTAua25vd25HZW5lKQp0eGRiIDwtIFR4RGIuTW11c2N1bHVzLlVDU0MubW0xMC5rbm93bkdlbmUKY29sdW1ucyh0eGRiKQoKQW5ub3RhdGlvbkRiaTo6c2VsZWN0KHR4ZGIsIGNvbHVtbnMgPSBjKCJFWE9OSUQiLCJFWE9OU1RBUlQiLCJFWE9OQ0hST00iKSwKICAgICAgICAgICAgICAgICAgICAgIGtleXM9IjIyMzczIiwgCiAgICAgICAgICAgICAgICAgICAgICBrZXl0eXBlID0gIkdFTkVJRCIpCmBgYAoKQWx0ZXJuYXRpdmVseSB3ZSBjYW4gZ3JhYiB0aGUgY29vcmRpbmF0ZXMgb2YgYWxsIGdlbmVzIGluIGEgc2luZ2xlIG9iamVjdCBhbmQgdGhlbiBzdWJzZXQgYWNjb3JkaW5nbHkuIFRoaXMgYWxsb3dzIHVzIHRvIHBlcmZvcm0gYWxsIGtpbmRzIG9mIHN1YnNldCBvcGVyYXRpb24gdXNpbmcgdGhlIGBHZW5vbWljRmVhdHVyZXNgIGZyYW1ld29yayBpbiBCaW9jb25kdWN0b3IuCgpgYGB7cn0KZXhvbnMgPC0gZXhvbnNCeSh0eGRiLCJnZW5lIikKZXhvbnMKZXhvbnNbWyIyMjM3MyJdXQpgYGAKCiMjIyBJbnRlcmFjdGl2ZSBncmFwaHMgYW5kIHRhYmxlcwoKSXQgaXMgb2Z0ZW4gdXNlZnVsIHRvIGJlIGFibGUgdG8gZXhwbG9yZSBvdXIgcmVzdWx0cyBpbiBhbiBpbnRlcmFjdGl2ZSBtYW5uZXI7IHNlYXJjaGluZyBmb3Igb3VyIGZhdm91cml0ZSBnZW5lcyBvZiBpbnRlcmVzdCBhbmQgcGxvdHRpbmcgb24tdGhlLWZseSB3aGV0aGVyIHRoZXkgYXJlIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQgaW4gb3VyIGRhdGFzZXQgb3Igbm90LgoKU3VjaCBhIHZpc3VhbGlzYXRpb24gaXMgcG9zc2libGUgd2l0aCB0aGUgW0dsaW1tYV0oaHR0cHM6Ly9hY2FkZW1pYy5vdXAuY29tL2Jpb2luZm9ybWF0aWNzL2FydGljbGUtbG9va3VwL2RvaS8xMC4xMDkzL2Jpb2luZm9ybWF0aWNzL2J0eDA5NCkgQmlvY29uZHVjdG9yIHBhY2thZ2UuIAoKSXQgdGFrZXMgb3VyIGBERVNlcTJgIHJlc3VsdHMgb2JqZWN0LCBhbm5vdGF0aW9uIHRhYmxlIGFuZCBub3JtYWxpemVkIGNvdW50cywgYW5kIHByb2R1Y2VzIGEgSFRNTCBwYWdlIGluY2x1ZGluZyBhIHNvcnRhYmxlIHJlc3VsdHMgdGFibGUsIE1BLXBsb3QgYW5kIHNjYXR0ZXIgcGxvdC4gUGFydGljdWxhciBnZW5lcyBjYW4gYmUgc2VhcmNoZWQgYW1vbmcgdGhlIHRhYmxlIGFuZCB0aGVpciBleHByZXNzaW9uIHBhdHRlcm5zIGNhbiBiZSBkaXNwbGF5ZWQuIEFsdGVybmF0aXZlbHkgd2UgY2FuIGNsaWNrIG9uIHBhcnRpY3VsYXIgcG9pbnQgaW4gdGhlIHBsb3QgYW5kIGRpc3BsYXkgdGhlaXIgc3RhdHMuCgoKCmBgYHtyfQoKcmVzdWx0cyA8LSByZXN1bHRzKGRlLm1mLGNvbnRyYXN0PWMoIlN0YXR1cyIsImxhY3RhdGlvbiIsInZpcmdpbiIpKQoKIyMgUmVwZWF0IHRoZSBhbm5vdGF0aW9uLCBhcyB0aGUgcHJldmlvdXMgYW5ub3RhdGlvbiB0YWJsZSB3YXMgY3JlYXRlZCB1c2luZyBhbiBvcmRlcmVkIHJlc3VsdHMgdGFibGUKCmFubm8gPC0gQW5ub3RhdGlvbkRiaTo6c2VsZWN0KG9yZy5NbS5lZy5kYixrZXlzPXJvd25hbWVzKHJlc3VsdHMpLAogICAgICAgICAgICAgIGNvbHVtbnM9YygiU1lNQk9MIiwiR0VORU5BTUUiKSwKICAgICAgICAgICAgICBrZXl0eXBlPSJFTlNFTUJMIikgJT4lIAogIGZpbHRlcighZHVwbGljYXRlZChFTlNFTUJMKSkKCmBgYAoKYGBge3J9CiMjIE1ha2Ugc3VyZSB3ZSBoYXZlIG5vcm1hbGlzZWQgY291bnRzIGJlZm9yZSBwcm9jZWVkaW5nCmRkcyA8LSBlc3RpbWF0ZVNpemVGYWN0b3JzKGRkcykKYGBgCgpgYGB7cn0KIyMgTG9hZCB0aGUgR2xpbW1hIHBhY2thZ2UgYW5kIGNyZWF0ZSB0aGUgcmVwb3J0CmxpYnJhcnkoR2xpbW1hKQpnbE1EUGxvdChyZXN1bHRzLAogICAgICAgICBhbm5vLAogICAgICAgICBncm91cHMgPSBjb2xEYXRhKGRkcykkU3RhdHVzLAogICAgICAgICBjb3VudHMgPSBjb3VudHMoZGRzLG5vcm1hbGl6ZWQ9VFJVRSksCiAgICAgICAgIHRyYW5zZm9ybSA9IFRSVUUsCiAgICAgICAgIHNpZGUubWFpbiA9ICJFTlNFTUJMIikKYGBgCgoK