Original Authors: Belinda Phipson, Anna Trigos, Matt Ritchie, Maria Doyle, Harriet Dashnow, Charity Law, Stephane Ballereau, Oscar Rueda, Ashley Sawle Based on the course RNAseq analysis in R delivered on May 11/12th 2016 and modified by Cancer Research Uk Cambridge Centre for the Functional Genomics Autumn School 2017

Differential expression with DESeq2

Now that we are happy that we have normalised the data and that the quality looks good, we can continue to testing for differentially expressed genes. There are a number of packages to analyse RNA-Seq data. Most people use DESeq2 or edgeR. We will use DESeq2 for the rest of this practical.

**First make sure we have all the objects and libraries loaded*

library(DESeq2)

Recap of pre-processing

The previous section walked-through the pre-processing and transformation of the count data. Here, for completeness, we list the minimal steps required to process the data prior to differential expression analysis.

Note that although we spent some time looking at the quality of our data , these steps are not required prior to performing differential expression so are not shown here. Remember, DESeq2 requires raw counts so the vst transformation is not shown as part of this basic protocol.

raw <- read_tsv("raw_data/selected_prostate_tcga_raw.tsv")
genes <- pull(raw,X)
cts <- as.matrix(raw[,-1])
rownames(cts) <- genes
sampleinfo <- read.delim("meta_data/sampleInfo_corrected.txt")
keep <- rowSums(assay(dds) >= 5) >= 5
dds <- dds[keep,]
dds <- DESeqDataSetFromMatrix(countData = cts, 
                                colData = sampleinfo,
                                design = ~Status)
dds <- dds[,-which(dds$ID %in% c("N7","N8","N9"))]
dds

We also have the output of the pre-processing section saved as an R object if you didn’t manage to complete these steps.

## Only run if you didn't complete the previous section on pre-processing
dds <- readRDS("Robjects/dds.rds")

Differential Expression with DESeq2

We have previously defined the test condition using the design argument when we created the object. This can be checked using the design function.

Typically we decide the design for the analysis when we create the DESeq2 objects, but it can be modified prior to the differential expression analysis

colData(dds)
DataFrame with 27 rows and 5 columns
                                               SampleName       ID gleason_score   Status     Batch
                                                 <factor> <factor>      <factor> <factor> <integer>
TCGA.CH.5753.01A.11R.1580.07 TCGA.CH.5753.01A.11R.1580.07     5753            G9   Tumour         1
TCGA.CH.5761.01A.11R.1580.07 TCGA.CH.5761.01A.11R.1580.07     5761            G9   Tumour         1
TCGA.EJ.5507.01A.01R.1580.07 TCGA.EJ.5507.01A.01R.1580.07     5507            G9   Tumour         1
TCGA.FC.A8O0.01A.41R.A37L.07 TCGA.FC.A8O0.01A.41R.A37L.07     A8O0            G6   Tumour         1
TCGA.G9.6371.01A.11R.1789.07 TCGA.G9.6371.01A.11R.1789.07     6371            G6   Tumour         1
...                                                   ...      ...           ...      ...       ...
TCGA.EJ.7327.11A.01R.2118.07 TCGA.EJ.7327.11A.01R.2118.07       N3        Normal   Normal         1
TCGA.G9.6356.11A.01R.1789.07 TCGA.G9.6356.11A.01R.1789.07       N4        Normal   Normal         1
TCGA.G9.6384.11A.01R.1858.07 TCGA.G9.6384.11A.01R.1858.07       N5        Normal   Normal         1
TCGA.G9.6499.11A.02R.1965.07 TCGA.G9.6499.11A.02R.1965.07       N6        Normal   Normal         2
TCGA.HC.8260.11A.01R.2263.07 TCGA.HC.8260.11A.01R.2263.07      N10        Normal   Normal         2
design(dds)
~Status

The function runs a couple of processing steps automatically to adjust for different library size and gene-wise variabiliy, which you can read about in the DESeq2 vignette.

Firstly apply the median ratio normalisation method to compensate for differences in library sizes

dds <- estimateSizeFactors(dds)

estimate the dispersion for each gene

dds <- estimateDispersions(dds)

Apply statistical testing based on the negative binomial distribution.

dds <- nbinomWaldTest(dds)

Fortunately, there is one convenient function that will apply the three steps

de_status <- DESeq(dds)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
-- replacing outliers and refitting for 947 genes
-- DESeq argument 'minReplicatesForReplace' = 7 
-- original counts are preserved in counts(dds)
estimating dispersions
fitting model and testing

The results of the analysis can be obtained using the results function and displayed to the screen. Each row is a particular gene measured in the study (i.e. all genes in the organism being studied) and each column reports some aspect of the differential expression analysis for that gene. Note that all genes are reported regardless of whether they are differentially-expressed or not. The results table is sorted according to the gene names rather than the test results.

results(de_status)

The output can be converted into a data frame and manipulated in the usual manner. It is recommended to use dplyr to manipulate the data frames with the standard set of operations detailed on the dplyr cheatsheet

  • select to pick which columns to display
  • filter to restrict the rows
  • mutate to add new variables to the data frame
  • arrange to order the data frame according to values of a column

However, dpylr does not like data frame that have rownames. We can use the rownames_to_column function from the tibble package to add an extra column that contains the Ensembl gene IDs.

library(dplyr)
library(tibble)
results_status <- as.data.frame(results(de_status)) %>% 
  rownames_to_column("GeneID")  

We can sort the rows by adjusted p-value and then print the first 10 rows.

arrange(results_status, padj) %>%  
  head(n=10)
    GeneID   baseMean log2FoldChange     lfcSE      stat       pvalue         padj
1  MIR4508  202.93938       6.704474 0.5325059 12.590422 2.383957e-36 4.576959e-32
2   SEMA5B 1258.54783       4.309721 0.3538565 12.179291 4.007503e-34 3.847003e-30
3    HSPA6 3148.75296      -7.457737 0.8235722 -9.055353 1.361266e-19 8.711650e-16
4   NKX2-3   77.94173       5.948540 0.7544276  7.884839 3.149416e-15 1.511641e-11
5    HOXC4  194.33661       4.485504 0.5905854  7.595013 3.077607e-14 1.181740e-10
6    HOXC5   51.46773       4.586805 0.6261784  7.325078 2.387600e-13 7.639921e-10
7    FOXD1  207.75221       4.835241 0.7122362  6.788817 1.130569e-11 3.100827e-08
8    HOXC6  501.89633       3.943275 0.6052434  6.515189 7.259817e-11 1.742265e-07
9    NETO2  471.09916       2.960785 0.4637273  6.384754 1.716737e-10 3.662181e-07
10    CST9   19.63986       5.628379 0.8847921  6.361245 2.001247e-10 3.676494e-07

We can sort the rows and then write the resulting data frame to a file.

arrange(results_status, padj) %>%
  write.csv("results/tumour_vs_normal_DESeq_all.csv")

Challenge 1

  1. Re-run the analysis to find differentially-expressed genes between the Gleason Grades 6 and 9
  2. Write a csv file that contains just the results for the genes that have a p-value less than 0.05 and a log2 fold change more than 1, or less than -1. HINT: So that we don’t overwrite our results so far, it may be convenient to create a new DESeqDataSet object for the new differential expression analysis.
dds_gleason <- dds
design(dds_gleason) <- ......

Changing the direction of the contrast

In this initial analyis DESeq2 has automatically decided which member of our sample groups to use as our baseline (Normal in this case) so that the log2 fold changes are reported with a positve value meaning higher expression in Tumour. If we want to change this behaviour we can change the contrast argument in the results function

## This should give the same as the table above
results(de_status, contrast=c("Status","Tumour","Normal"))
## Changing the direction of the contrast
results(de_status, contrast=c("Status","Normal","Tumour"))

If we change to performing differential expression analysis on the gleason_score variable then there are various contrasts that can be made; gleason 6 vs Normal, Gleason 9 vs Normal etc. When the results function is run the table that is displayed is for the contrast Normal vs G6. The resultsNames function can tell us which other contrasts we can access.

dds_gleason <- dds
design(dds_gleason) <- ~gleason_score
de_gleason <- DESeq(dds_gleason)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
-- replacing outliers and refitting for 577 genes
-- DESeq argument 'minReplicatesForReplace' = 7 
-- original counts are preserved in counts(dds)
estimating dispersions
fitting model and testing
resultsNames(de_gleason)
[1] "Intercept"                  "gleason_score_G9_vs_G6"     "gleason_score_Normal_vs_G6"
results_gleason <- data.frame(results(de_gleason))

Intersecting gene lists

A venn diagram is a common way of visualising the overlap between two genelists. We need to create a data frame where each column indicates whether each gene is differentially expressed in a particular contrast or not. To create such columns we can do a logical test on the adjusted p-values from our results tables.

venn_data <- data.frame(CellType = results_status$padj<0.05,
                        Gleason = results_gleason$padj < 0.05)
library(limma)
vennDiagram(venn_data)

Fitting alternative models to the data

DESEq2 allows for more complicated models to be fit to the data. For guidance on how to fit more complicated models you can consult the DESeq2 vignette, the limma user guide or the Bioconductor mailing list.

In particular, DESeq2 allows multi-factor models which can account for other sources of variation in the data such as batches or gender.

Lets suppose that we wanted the different between virgin and lactatin individuals, but controlling for Batch. The main assumption being that the effect of Status is the same regardless of Batch The design for such an analysis would be:-

dds_mf <- dds
design(dds_mf) <- ~Batch+Status
de_mf <- DESeq(dds.mf)
estimating size factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
-- replacing outliers and refitting for 600 genes
-- DESeq argument 'minReplicatesForReplace' = 7 
-- original counts are preserved in counts(dds)
estimating dispersions
fitting model and testing
results_mf <- results(de_mf,contrast=c("Status","Tumour","Normal"))
results_mf
log2 fold change (MLE): Status Tumour vs Normal 
Wald test p-value: Status Tumour vs Normal 
DataFrame with 23368 rows and 6 columns
                    baseMean      log2FoldChange             lfcSE                stat              pvalue
                   <numeric>           <numeric>         <numeric>           <numeric>           <numeric>
1/2-SBSRNA4 53.0443987895977     0.2832355361628 0.226779705469357    1.24894569192864   0.211684937966744
A1BG        221.123479031816   -1.17471447179457 0.508517726511583   -2.31007575655834  0.0208839603703524
A1BG-AS1    29.7317978744033   -1.32549411546739 0.474210774088385   -2.79515816150635 0.00518743297291225
A1CF        7.72356776697792    1.43903927022599 0.730103467956311    1.97100730702473  0.0487230400302148
A2LD1       400.322513053529   0.474744895157193 0.199951414788572    2.37430125542841   0.017582198742755
...                      ...                 ...               ...                 ...                 ...
ZYG11B      2579.01965251234  -0.299743258592344 0.169578593358214   -1.76757721983914  0.0771316091131724
ZYX         8071.18437115632  -0.387691129714366 0.240282043951644   -1.61348356847001   0.106639490283516
ZZEF1       3233.23162325016  -0.251943528887863 0.186782910984024    -1.3488574921579    0.17738274398432
ZZZ3        2204.88768617892   0.149833112009978 0.151518264585969   0.988878221509495   0.322722727159437
tAKR          1.548947012253 -0.0606553024026499 0.781497440271342 -0.0776142048291161   0.938134942648593
                          padj
                     <numeric>
1/2-SBSRNA4  0.404505048256582
A1BG        0.0867887267993926
A1BG-AS1    0.0326402675538388
A1CF         0.154263757037365
A2LD1       0.0765231802955422
...                        ...
ZYG11B       0.209847384936015
ZYX            0.2601979804504
ZZEF1          0.3615648129121
ZZZ3         0.525498455619574
tAKR         0.968830992496522

Exporting normalized counts

The DESeq workflow applies median of ratios normalization that accounts for differences in sequencing depth between samples. The user does not usually need to run this step. However, if you want a matrix of counts for some application outside of Bioconductor the values can be extracted from the dds object.

dds <- estimateSizeFactors(dds) 
countMatrix <-counts(dds, normalized=TRUE)
write.csv(countMatrix,file="processed_data/normalized_counts.csv")

We can also save the DESeq2 results so that we don’t have to repeat the analysis

saveRDS(de_status, file="Robjects/de_status.rds")
saveRDS(de_gleason, file="Robjects/de_gleason.rds")
LS0tCnRpdGxlOiAiUk5BLXNlcSBhbmFseXNpcyBpbiBSIgpzdWJ0aXRsZTogIkRpZmZlcmVudGlhbCBFeHByZXNzaW9uIG9mIFJOQS1zZXEgZGF0YSIKYXV0aG9yOiAiTWFyayBEdW5uaW5nIgpkYXRlOiBGZWJydWFyeSAyMDIwCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgdG9jOiB5ZXMKICAgIHRvY19mbG9hdDogeWVzCiAgaHRtbF9kb2N1bWVudDoKICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwptaW51dGVzOiAzMDAKbGF5b3V0OiBwYWdlCi0tLQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSkKYGBgCgoqKk9yaWdpbmFsIEF1dGhvcnM6IEJlbGluZGEgUGhpcHNvbiwgQW5uYSBUcmlnb3MsIE1hdHQgUml0Y2hpZSwgTWFyaWEgRG95bGUsIEhhcnJpZXQgRGFzaG5vdywgQ2hhcml0eSBMYXcqKiwgKipTdGVwaGFuZSBCYWxsZXJlYXUsIE9zY2FyIFJ1ZWRhLCBBc2hsZXkgU2F3bGUqKgpCYXNlZCBvbiB0aGUgY291cnNlIFtSTkFzZXEgYW5hbHlzaXMgaW4gUl0oaHR0cDovL2NvbWJpbmUtYXVzdHJhbGlhLmdpdGh1Yi5pby8yMDE2LTA1LTExLVJOQXNlcS8pIGRlbGl2ZXJlZCBvbiBNYXkgMTEvMTJ0aCAyMDE2IGFuZCBtb2RpZmllZCBieSBDYW5jZXIgUmVzZWFyY2ggVWsgQ2FtYnJpZGdlIENlbnRyZSBmb3IgdGhlIFtGdW5jdGlvbmFsIEdlbm9taWNzIEF1dHVtbiBTY2hvb2wgMjAxN10oaHR0cHM6Ly9iaW9pbmZvcm1hdGljcy1jb3JlLXNoYXJlZC10cmFpbmluZy5naXRodWIuaW8vY3J1ay1hdXR1bW4tc2Nob29sLTIwMTcvKQoKIyMgUmVzb3VyY2VzIGFuZCBkYXRhIGZpbGVzCgpUaGlzIG1hdGVyaWFsIGhhcyBiZWVuIGNyZWF0ZWQgdXNpbmcgdGhlIGZvbGxvd2luZyByZXNvdXJjZXM6ICAKCgotIGh0dHA6Ly9tb25hc2hiaW9pbmZvcm1hdGljc3BsYXRmb3JtLmdpdGh1Yi5pby9STkFzZXEtREUtYW5hbHlzaXMtd2l0aC1SLzk5LVJOQXNlcV9ERV9hbmFseXNpc193aXRoX1IuaHRtbCAgCi0gaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sCi0gaHR0cHM6Ly9iaW9jb25kdWN0b3IuZ2l0aHViLmlvL0Jpb2NXb3Jrc2hvcHMvcm5hLXNlcS1kYXRhLWFuYWx5c2lzLXdpdGgtZGVzZXEyLmh0bWwKCgoKIyMgRGlmZmVyZW50aWFsIGV4cHJlc3Npb24gd2l0aCBgREVTZXEyYAoKTm93IHRoYXQgd2UgYXJlIGhhcHB5IHRoYXQgd2UgaGF2ZSBub3JtYWxpc2VkIHRoZSBkYXRhIGFuZCB0aGF0IHRoZSBxdWFsaXR5IGxvb2tzIGdvb2QsIHdlIGNhbiBjb250aW51ZSB0byB0ZXN0aW5nIGZvciBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMuIFRoZXJlIGFyZSBhIG51bWJlciBvZiBwYWNrYWdlcyB0byBhbmFseXNlIFJOQS1TZXEgZGF0YS4gTW9zdCBwZW9wbGUgdXNlIGBERVNlcTJgIG9yIGBlZGdlUmAuIFdlIHdpbGwgdXNlIGBERVNlcTJgIGZvciB0aGUgcmVzdCBvZiB0aGlzIHByYWN0aWNhbC4KCioqRmlyc3QgbWFrZSBzdXJlIHdlIGhhdmUgYWxsIHRoZSBvYmplY3RzIGFuZCBsaWJyYXJpZXMgbG9hZGVkKgoKYGBge3J9CmxpYnJhcnkoREVTZXEyKQoKYGBgCgoKIyMjIFJlY2FwIG9mIHByZS1wcm9jZXNzaW5nCgpUaGUgcHJldmlvdXMgc2VjdGlvbiB3YWxrZWQtdGhyb3VnaCB0aGUgcHJlLXByb2Nlc3NpbmcgYW5kIHRyYW5zZm9ybWF0aW9uIG9mIHRoZSBjb3VudCBkYXRhLiBIZXJlLCBmb3IgY29tcGxldGVuZXNzLCB3ZSBsaXN0IHRoZSBtaW5pbWFsIHN0ZXBzIHJlcXVpcmVkIHRvIHByb2Nlc3MgdGhlIGRhdGEgcHJpb3IgdG8gZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMuCgpOb3RlIHRoYXQgYWx0aG91Z2ggd2Ugc3BlbnQgc29tZSB0aW1lIGxvb2tpbmcgYXQgdGhlIHF1YWxpdHkgb2Ygb3VyIGRhdGEgLCB0aGVzZSBzdGVwcyBhcmUgbm90IHJlcXVpcmVkIHByaW9yIHRvIHBlcmZvcm1pbmcgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gc28gYXJlIG5vdCBzaG93biBoZXJlLiBSZW1lbWJlciwgYERFU2VxMmAgW3JlcXVpcmVzIHJhdyBjb3VudHNdKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCN3aHktdW4tbm9ybWFsaXplZC1jb3VudHMpIHNvIHRoZSBgdnN0YCB0cmFuc2Zvcm1hdGlvbiBpcyBub3Qgc2hvd24gYXMgcGFydCBvZiB0aGlzIGJhc2ljIHByb3RvY29sLgoKYGBge3IgZXZhbD1GQUxTRX0KcmF3IDwtIHJlYWRfdHN2KCJyYXdfZGF0YS9zZWxlY3RlZF9wcm9zdGF0ZV90Y2dhX3Jhdy50c3YiKQpnZW5lcyA8LSBwdWxsKHJhdyxYKQpjdHMgPC0gYXMubWF0cml4KHJhd1ssLTFdKQpyb3duYW1lcyhjdHMpIDwtIGdlbmVzCnNhbXBsZWluZm8gPC0gcmVhZC5kZWxpbSgibWV0YV9kYXRhL3NhbXBsZUluZm9fY29ycmVjdGVkLnR4dCIpCmtlZXAgPC0gcm93U3Vtcyhhc3NheShkZHMpID49IDUpID49IDUKZGRzIDwtIGRkc1trZWVwLF0KZGRzIDwtIERFU2VxRGF0YVNldEZyb21NYXRyaXgoY291bnREYXRhID0gY3RzLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xEYXRhID0gc2FtcGxlaW5mbywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkZXNpZ24gPSB+U3RhdHVzKQpkZHMgPC0gZGRzWywtd2hpY2goZGRzJElEICVpbiUgYygiTjciLCJOOCIsIk45IikpXQpkZHMKYGBgCgpXZSBhbHNvIGhhdmUgdGhlIG91dHB1dCBvZiB0aGUgcHJlLXByb2Nlc3Npbmcgc2VjdGlvbiBzYXZlZCBhcyBhbiBSIG9iamVjdCBpZiB5b3UgZGlkbid0IG1hbmFnZSB0byBjb21wbGV0ZSB0aGVzZSBzdGVwcy4KCmBgYHtyIGV2YWw9RkFMU0V9CiMjIE9ubHkgcnVuIGlmIHlvdSBkaWRuJ3QgY29tcGxldGUgdGhlIHByZXZpb3VzIHNlY3Rpb24gb24gcHJlLXByb2Nlc3NpbmcKZGRzIDwtIHJlYWRSRFMoIlJvYmplY3RzL2Rkcy5yZHMiKQpgYGAKCgoKIyMgRGlmZmVyZW50aWFsIEV4cHJlc3Npb24gd2l0aCBERVNlcTIKCldlIGhhdmUgcHJldmlvdXNseSBkZWZpbmVkIHRoZSB0ZXN0IGNvbmRpdGlvbiB1c2luZyB0aGUgYGRlc2lnbmAgYXJndW1lbnQgd2hlbiB3ZSBjcmVhdGVkIHRoZSBvYmplY3QuIFRoaXMgY2FuIGJlIGNoZWNrZWQgdXNpbmcgdGhlIGBkZXNpZ25gIGZ1bmN0aW9uLgoKVHlwaWNhbGx5IHdlIGRlY2lkZSB0aGUgZGVzaWduIGZvciB0aGUgYW5hbHlzaXMgd2hlbiB3ZSBjcmVhdGUgdGhlIERFU2VxMiBvYmplY3RzLCBidXQgaXQgY2FuIGJlIG1vZGlmaWVkIHByaW9yIHRvIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcwoKYGBge3J9CmNvbERhdGEoZGRzKQpkZXNpZ24oZGRzKQpgYGAKCgoKVGhlIGZ1bmN0aW9uIHJ1bnMgYSBjb3VwbGUgb2YgcHJvY2Vzc2luZyBzdGVwcyBhdXRvbWF0aWNhbGx5IHRvIGFkanVzdCBmb3IgZGlmZmVyZW50IGxpYnJhcnkgc2l6ZSBhbmQgZ2VuZS13aXNlIHZhcmlhYmlsaXksIHdoaWNoIHlvdSBjYW4gcmVhZCBhYm91dCBpbiB0aGUgW0RFU2VxMiB2aWduZXR0ZV0oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI3RoZS1kZXNlcTItbW9kZWwpLgoKRmlyc3RseSBhcHBseSB0aGUgbWVkaWFuIHJhdGlvIG5vcm1hbGlzYXRpb24gbWV0aG9kIHRvIGNvbXBlbnNhdGUgZm9yIGRpZmZlcmVuY2VzIGluIGxpYnJhcnkgc2l6ZXMKCmBgYHtyIGV2YWw9RkFMU0V9CmRkcyA8LSBlc3RpbWF0ZVNpemVGYWN0b3JzKGRkcykKCmBgYAoKZXN0aW1hdGUgdGhlIGRpc3BlcnNpb24gZm9yIGVhY2ggZ2VuZQoKYGBge3IgZXZhbD1GQUxTRX0KZGRzIDwtIGVzdGltYXRlRGlzcGVyc2lvbnMoZGRzKQpgYGAKCkFwcGx5IHN0YXRpc3RpY2FsIHRlc3RpbmcgYmFzZWQgb24gdGhlIG5lZ2F0aXZlIGJpbm9taWFsIGRpc3RyaWJ1dGlvbi4KCmBgYHtyIGV2YWw9RkFMU0V9CmRkcyA8LSBuYmlub21XYWxkVGVzdChkZHMpCmBgYAoKRm9ydHVuYXRlbHksIHRoZXJlIGlzIG9uZSBjb252ZW5pZW50IGZ1bmN0aW9uIHRoYXQgd2lsbCBhcHBseSB0aGUgdGhyZWUgc3RlcHMKCmBgYHtyfQpkZV9zdGF0dXMgPC0gREVTZXEoZGRzKQoKYGBgCgoKClRoZSByZXN1bHRzIG9mIHRoZSBhbmFseXNpcyBjYW4gYmUgb2J0YWluZWQgdXNpbmcgdGhlIGByZXN1bHRzYCBmdW5jdGlvbiBhbmQgZGlzcGxheWVkIHRvIHRoZSBzY3JlZW4uIEVhY2ggcm93IGlzIGEgcGFydGljdWxhciBnZW5lIG1lYXN1cmVkIGluIHRoZSBzdHVkeSAoaS5lLiBhbGwgZ2VuZXMgaW4gdGhlIG9yZ2FuaXNtIGJlaW5nIHN0dWRpZWQpIGFuZCBlYWNoIGNvbHVtbiByZXBvcnRzIHNvbWUgYXNwZWN0IG9mIHRoZSBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcyBmb3IgdGhhdCBnZW5lLiBOb3RlIHRoYXQgKiphbGwqKiBnZW5lcyBhcmUgcmVwb3J0ZWQgcmVnYXJkbGVzcyBvZiB3aGV0aGVyIHRoZXkgYXJlIGRpZmZlcmVudGlhbGx5LWV4cHJlc3NlZCBvciBub3QuIFRoZSByZXN1bHRzIHRhYmxlIGlzIHNvcnRlZCBhY2NvcmRpbmcgdG8gdGhlIGdlbmUgbmFtZXMgcmF0aGVyIHRoYW4gdGhlIHRlc3QgcmVzdWx0cy4KCmBgYHtyIGV2YWw9RkFMU0V9CnJlc3VsdHMoZGVfc3RhdHVzKQpgYGAKClRoZSBvdXRwdXQgY2FuIGJlIGNvbnZlcnRlZCBpbnRvIGEgZGF0YSBmcmFtZSBhbmQgbWFuaXB1bGF0ZWQgaW4gdGhlIHVzdWFsIG1hbm5lci4gSXQgaXMgcmVjb21tZW5kZWQgdG8gdXNlIGBkcGx5cmAgdG8gbWFuaXB1bGF0ZSB0aGUgZGF0YSBmcmFtZXMgd2l0aCB0aGUgc3RhbmRhcmQgc2V0IG9mIG9wZXJhdGlvbnMgZGV0YWlsZWQgb24gdGhlIFtkcGx5ciBjaGVhdHNoZWV0XShodHRwczovL3d3dy5yc3R1ZGlvLmNvbS93cC1jb250ZW50L3VwbG9hZHMvMjAxNS8wMi9kYXRhLXdyYW5nbGluZy1jaGVhdHNoZWV0LnBkZikKCi0gYHNlbGVjdGAgdG8gcGljayB3aGljaCBjb2x1bW5zIHRvIGRpc3BsYXkKLSBgZmlsdGVyYCB0byByZXN0cmljdCB0aGUgcm93cwotIGBtdXRhdGVgIHRvIGFkZCBuZXcgdmFyaWFibGVzIHRvIHRoZSBkYXRhIGZyYW1lCi0gYGFycmFuZ2VgIHRvIG9yZGVyIHRoZSBkYXRhIGZyYW1lIGFjY29yZGluZyB0byB2YWx1ZXMgb2YgYSBjb2x1bW4KCkhvd2V2ZXIsIGBkcHlscmAgZG9lcyBub3QgbGlrZSBkYXRhIGZyYW1lIHRoYXQgaGF2ZSByb3duYW1lcy4gV2UgY2FuIHVzZSB0aGUgYHJvd25hbWVzX3RvX2NvbHVtbmAgZnVuY3Rpb24gZnJvbSB0aGUgYHRpYmJsZWAgcGFja2FnZSB0byBhZGQgYW4gZXh0cmEgY29sdW1uIHRoYXQgY29udGFpbnMgdGhlIEVuc2VtYmwgZ2VuZSBJRHMuCgoKYGBge3J9CmxpYnJhcnkoZHBseXIpCmxpYnJhcnkodGliYmxlKQpyZXN1bHRzX3N0YXR1cyA8LSBhcy5kYXRhLmZyYW1lKHJlc3VsdHMoZGVfc3RhdHVzKSkgJT4lIAogIHJvd25hbWVzX3RvX2NvbHVtbigiR2VuZUlEIikgIAoKYGBgCgpXZSBjYW4gc29ydCB0aGUgcm93cyBieSBhZGp1c3RlZCBwLXZhbHVlIGFuZCB0aGVuIHByaW50IHRoZSBmaXJzdCAxMCByb3dzLgoKYGBge3J9CmFycmFuZ2UocmVzdWx0c19zdGF0dXMsIHBhZGopICU+JSAgCiAgaGVhZChuPTEwKQpgYGAKCldlIGNhbiBzb3J0IHRoZSByb3dzIGFuZCB0aGVuIHdyaXRlIHRoZSByZXN1bHRpbmcgZGF0YSBmcmFtZSB0byBhIGZpbGUuCgpgYGB7cn0KYXJyYW5nZShyZXN1bHRzX3N0YXR1cywgcGFkaikgJT4lCiAgd3JpdGUuY3N2KCJyZXN1bHRzL3R1bW91cl92c19ub3JtYWxfREVTZXFfYWxsLmNzdiIpCmBgYAoKCgoKPiAjIyBDaGFsbGVuZ2UgMSB7LmNoYWxsZW5nZX0KPgo+IDEuIFJlLXJ1biB0aGUgYW5hbHlzaXMgdG8gZmluZCBkaWZmZXJlbnRpYWxseS1leHByZXNzZWQgZ2VuZXMgYmV0d2VlbiB0aGUgR2xlYXNvbiBHcmFkZXMgNiBhbmQgOQo+IDIuIFdyaXRlIGEgY3N2IGZpbGUgdGhhdCBjb250YWlucyBqdXN0IHRoZSByZXN1bHRzIGZvciB0aGUgZ2VuZXMgdGhhdCBoYXZlIGEgcC12YWx1ZSBsZXNzIHRoYW4gMC4wNSBhbmQgYSBsb2cyIGZvbGQgY2hhbmdlIG1vcmUgdGhhbiAxLCBvciBsZXNzIHRoYW4gLTEuCj4gSElOVDogU28gdGhhdCB3ZSBkb24ndCBvdmVyd3JpdGUgb3VyIHJlc3VsdHMgc28gZmFyLCBpdCBtYXkgYmUgY29udmVuaWVudCB0byBjcmVhdGUgYSBuZXcgYERFU2VxRGF0YVNldGAgb2JqZWN0IGZvciB0aGUgbmV3IGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzLgoKYGBge3IgZXZhbD1GQUxTRX0KZGRzX2dsZWFzb24gPC0gZGRzCmRlc2lnbihkZHNfZ2xlYXNvbikgPC0gLi4uLi4uCmBgYAoKIyMjIENoYW5naW5nIHRoZSBkaXJlY3Rpb24gb2YgdGhlIGNvbnRyYXN0CgpJbiB0aGlzIGluaXRpYWwgYW5hbHlpcyBgREVTZXEyYCBoYXMgYXV0b21hdGljYWxseSBkZWNpZGVkIHdoaWNoIG1lbWJlciBvZiBvdXIgc2FtcGxlIGdyb3VwcyB0byB1c2UgYXMgb3VyIGJhc2VsaW5lIChgTm9ybWFsYCBpbiB0aGlzIGNhc2UpIHNvIHRoYXQgdGhlIGxvZzIgZm9sZCBjaGFuZ2VzIGFyZSByZXBvcnRlZCB3aXRoIGEgcG9zaXR2ZSB2YWx1ZSBtZWFuaW5nIGhpZ2hlciBleHByZXNzaW9uIGluIGBUdW1vdXJgLiBJZiB3ZSB3YW50IHRvIGNoYW5nZSB0aGlzIGJlaGF2aW91ciB3ZSBjYW4gY2hhbmdlIHRoZSBgY29udHJhc3RgIGFyZ3VtZW50IGluIHRoZSBgcmVzdWx0c2AgZnVuY3Rpb24KCgpgYGB7ciBldmFsPUZBTFNFfQojIyBUaGlzIHNob3VsZCBnaXZlIHRoZSBzYW1lIGFzIHRoZSB0YWJsZSBhYm92ZQpyZXN1bHRzKGRlX3N0YXR1cywgY29udHJhc3Q9YygiU3RhdHVzIiwiVHVtb3VyIiwiTm9ybWFsIikpCiMjIENoYW5naW5nIHRoZSBkaXJlY3Rpb24gb2YgdGhlIGNvbnRyYXN0CnJlc3VsdHMoZGVfc3RhdHVzLCBjb250cmFzdD1jKCJTdGF0dXMiLCJOb3JtYWwiLCJUdW1vdXIiKSkKCmBgYAoKSWYgd2UgY2hhbmdlIHRvIHBlcmZvcm1pbmcgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMgb24gdGhlIGBnbGVhc29uX3Njb3JlYCB2YXJpYWJsZSB0aGVuIHRoZXJlIGFyZSB2YXJpb3VzIGNvbnRyYXN0cyB0aGF0IGNhbiBiZSBtYWRlOyBgZ2xlYXNvbiA2YCB2cyBgTm9ybWFsYCwgYEdsZWFzb24gOWAgdnMgYE5vcm1hbGAgZXRjLiBXaGVuIHRoZSBgcmVzdWx0c2AgZnVuY3Rpb24gaXMgcnVuIHRoZSB0YWJsZSB0aGF0IGlzIGRpc3BsYXllZCBpcyBmb3IgdGhlIGNvbnRyYXN0IGBOb3JtYWwgdnMgRzZgLiBUaGUgYHJlc3VsdHNOYW1lc2AgZnVuY3Rpb24gY2FuIHRlbGwgdXMgd2hpY2ggb3RoZXIgY29udHJhc3RzIHdlIGNhbiBhY2Nlc3MuCgoKYGBge3J9CmRkc19nbGVhc29uIDwtIGRkcwpkZXNpZ24oZGRzX2dsZWFzb24pIDwtIH5nbGVhc29uX3Njb3JlCmRlX2dsZWFzb24gPC0gREVTZXEoZGRzX2dsZWFzb24pCnJlc3VsdHNOYW1lcyhkZV9nbGVhc29uKQpyZXN1bHRzX2dsZWFzb24gPC0gZGF0YS5mcmFtZShyZXN1bHRzKGRlX2dsZWFzb24pKQpgYGAKCiMjIyBJbnRlcnNlY3RpbmcgZ2VuZSBsaXN0cwoKQSB2ZW5uIGRpYWdyYW0gaXMgYSBjb21tb24gd2F5IG9mIHZpc3VhbGlzaW5nIHRoZSBvdmVybGFwIGJldHdlZW4gdHdvIGdlbmVsaXN0cy4gV2UgbmVlZCB0byBjcmVhdGUgYSBkYXRhIGZyYW1lIHdoZXJlIGVhY2ggY29sdW1uIGluZGljYXRlcyB3aGV0aGVyIGVhY2ggZ2VuZSBpcyBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgaW4gYSBwYXJ0aWN1bGFyIGNvbnRyYXN0IG9yIG5vdC4gVG8gY3JlYXRlIHN1Y2ggY29sdW1ucyB3ZSBjYW4gZG8gYSBsb2dpY2FsIHRlc3Qgb24gdGhlIGFkanVzdGVkIHAtdmFsdWVzIGZyb20gb3VyIHJlc3VsdHMgdGFibGVzLgoKCmBgYHtyIG1lc3NhZ2U9RkFMU0V9CnZlbm5fZGF0YSA8LSBkYXRhLmZyYW1lKENlbGxUeXBlID0gcmVzdWx0c19zdGF0dXMkcGFkajwwLjA1LAogICAgICAgICAgICAgICAgICAgICAgICBHbGVhc29uID0gcmVzdWx0c19nbGVhc29uJHBhZGogPCAwLjA1KQpsaWJyYXJ5KGxpbW1hKQp2ZW5uRGlhZ3JhbSh2ZW5uX2RhdGEpCmBgYAoKCgoKCiMjIyBGaXR0aW5nIGFsdGVybmF0aXZlIG1vZGVscyB0byB0aGUgZGF0YQoKYERFU0VxMmAgYWxsb3dzIGZvciBtb3JlIGNvbXBsaWNhdGVkIG1vZGVscyB0byBiZSBmaXQgdG8gdGhlIGRhdGEuIEZvciBndWlkYW5jZSBvbiBob3cgdG8gZml0IG1vcmUgY29tcGxpY2F0ZWQgbW9kZWxzIHlvdSBjYW4gY29uc3VsdCB0aGUgW0RFU2VxMiB2aWduZXR0ZV0oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sKSwgdGhlIFtsaW1tYSB1c2VyIGd1aWRlXShodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL2xpbW1hL2luc3QvZG9jL3VzZXJzZ3VpZGUucGRmKSBvciB0aGUgQmlvY29uZHVjdG9yIG1haWxpbmcgbGlzdC4gCgpJbiBwYXJ0aWN1bGFyLCBERVNlcTIgYWxsb3dzIFttdWx0aS1mYWN0b3IgbW9kZWxzXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwjbXVsdGktZmFjdG9yLWRlc2lnbnMpIHdoaWNoIGNhbiBhY2NvdW50IGZvciBvdGhlciBzb3VyY2VzIG9mIHZhcmlhdGlvbiBpbiB0aGUgZGF0YSBzdWNoIGFzIGJhdGNoZXMgb3IgZ2VuZGVyLgoKTGV0cyBzdXBwb3NlIHRoYXQgd2Ugd2FudGVkIHRoZSBkaWZmZXJlbnQgYmV0d2VlbiB2aXJnaW4gYW5kIGxhY3RhdGluIGluZGl2aWR1YWxzLCBidXQgY29udHJvbGxpbmcgZm9yIGBCYXRjaGAuIFRoZSBtYWluIGFzc3VtcHRpb24gYmVpbmcgdGhhdCB0aGUgZWZmZWN0IG9mIGBTdGF0dXNgIGlzIHRoZSBzYW1lIHJlZ2FyZGxlc3Mgb2YgYEJhdGNoYCBUaGUgZGVzaWduIGZvciBzdWNoIGFuIGFuYWx5c2lzIHdvdWxkIGJlOi0KCmBgYHtyfQpkZHNfbWYgPC0gZGRzCmRlc2lnbihkZHNfbWYpIDwtIH5CYXRjaCtTdGF0dXMKZGVfbWYgPC0gREVTZXEoZGRzLm1mKQpyZXN1bHRzX21mIDwtIHJlc3VsdHMoZGVfbWYsY29udHJhc3Q9YygiU3RhdHVzIiwiVHVtb3VyIiwiTm9ybWFsIikpCgpgYGAKCgojIyMgRXhwb3J0aW5nIG5vcm1hbGl6ZWQgY291bnRzCgpUaGUgYERFU2VxYCB3b3JrZmxvdyBhcHBsaWVzICptZWRpYW4gb2YgcmF0aW9zIG5vcm1hbGl6YXRpb24qIHRoYXQgYWNjb3VudHMgZm9yIGRpZmZlcmVuY2VzIGluIHNlcXVlbmNpbmcgZGVwdGggYmV0d2VlbiBzYW1wbGVzLiBUaGUgdXNlciBkb2VzIG5vdCB1c3VhbGx5IG5lZWQgdG8gcnVuIHRoaXMgc3RlcC4gSG93ZXZlciwgaWYgeW91IHdhbnQgYSBtYXRyaXggb2YgY291bnRzIGZvciBzb21lIGFwcGxpY2F0aW9uIG91dHNpZGUgb2YgQmlvY29uZHVjdG9yIHRoZSB2YWx1ZXMgY2FuIGJlIGV4dHJhY3RlZCBmcm9tIHRoZSBgZGRzYCBvYmplY3QuCgpgYGB7cn0KZGRzIDwtIGVzdGltYXRlU2l6ZUZhY3RvcnMoZGRzKSAKY291bnRNYXRyaXggPC1jb3VudHMoZGRzLCBub3JtYWxpemVkPVRSVUUpCndyaXRlLmNzdihjb3VudE1hdHJpeCxmaWxlPSJwcm9jZXNzZWRfZGF0YS9ub3JtYWxpemVkX2NvdW50cy5jc3YiKQpgYGAKCldlIGNhbiBhbHNvIHNhdmUgdGhlIERFU2VxMiByZXN1bHRzIHNvIHRoYXQgd2UgZG9uJ3QgaGF2ZSB0byByZXBlYXQgdGhlIGFuYWx5c2lzCgpgYGB7cn0Kc2F2ZVJEUyhkZV9zdGF0dXMsIGZpbGU9IlJvYmplY3RzL2RlX3N0YXR1cy5yZHMiKQpzYXZlUkRTKGRlX2dsZWFzb24sIGZpbGU9IlJvYmplY3RzL2RlX2dsZWFzb24ucmRzIikKYGBgCgo=