Original Authors: Belinda Phipson, Anna Trigos, Matt Ritchie, Maria Doyle, Harriet Dashnow, Charity Law, Stephane Ballereau, Oscar Rueda, Ashley Sawle Based on the course RNAseq analysis in R delivered on May 11/12th 2016 and modified by Cancer Research Uk Cambridge Centre for the Functional Genomics Autumn School 2017

Differential expression with DESeq2

Now that we are happy that we have normalised the data and that the quality looks good, we can continue to testing for differentially expressed genes. There are a number of packages to analyse RNA-Seq data. Most people use DESeq2 or edgeR. We will use DESeq2 for the rest of this practical.

**First make sure we have all the objects and libraries loaded*

library(DESeq2)
library(tximport)

Recap of pre-processing

The previous section walked-through the pre-processing and transformation of the count data. Here, for completeness, we list the minimal steps required to process the data prior to differential expression analysis.

Note that although we spent some time looking at the quality of our data , these steps are not required prior to performing differential expression so are not shown here. Remember, DESeq2 requires raw counts so the vst transformation is not shown as part of this basic protocol.

dirs <- list.files(file.path("salmon_quant/"))
quant_files <- paste0("salmon_quant/",dirs,"/quant.sf.gz")
names(quant_files) <- dirs

tx2gene <- read.csv("tx2gene.csv")

txi <- tximport(quant_files,type="salmon",tx2gene = tx2gene,ignoreTxVersion = TRUE)

sampleinfo <- read.delim("meta_data/sampleInfo_corrected.txt")
rownames(sampleinfo) <- sampleinfo$run

dds <- DESeqDataSetFromTximport(txi, 
                                colData = sampleinfo,
                                design <- ~CellType)
keep <- rowSums(assay(dds) >= 5) >= 2
dds <- dds[keep,]

We also have the output of the pre-processing section saved as an R object if you didn’t manage to complete these steps.

## Only run if you didn't complete the previous section on pre-processing
load("Robjects/preprocessing.Rdata")

Differential Expression with DESeq2

We have previously defined the test condition using the design argument when we created the object. This can be checked using the design function.

Typically we decide the design for the analysis when we create the DESeq2 objects, but it can be modified prior to the differential expression analysis

colData(dds)
DataFrame with 12 rows and 4 columns
                  run     Name CellType    Status
             <factor> <factor> <factor>  <factor>
SRR1552444 SRR1552444  MCL1-LA  luminal    virgin
SRR1552445 SRR1552445  MCL1-LB  luminal    virgin
SRR1552446 SRR1552446  MCL1-LC  luminal pregnancy
SRR1552447 SRR1552447  MCL1-LD  luminal pregnancy
SRR1552448 SRR1552448  MCL1-LE  luminal lactation
...               ...      ...      ...       ...
SRR1552451 SRR1552451  MCL1-DH    basal    virgin
SRR1552452 SRR1552452  MCL1-DI    basal pregnancy
SRR1552453 SRR1552453  MCL1-DJ    basal pregnancy
SRR1552454 SRR1552454  MCL1-DK    basal lactation
SRR1552455 SRR1552455  MCL1-DL    basal lactation
design(dds) <- ~CellType

The function runs a couple of processing steps automatically to adjust for different library size and gene-wise variabiliy, which you can read about in the DESeq2 vignette.

Firstly apply the median ratio normalisation method to compensate for differences in library sizes

dds <- estimateSizeFactors(dds)

estimate the dispersion for each gene

dds <- estimateDispersions(dds)

Apply statistical testing based on the negative binomial distribution.

dds <- nbinomWaldTest(dds)

Fortunately, there is one convenient function that will apply the three steps

de.cellType <- DESeq(dds)
using pre-existing normalization factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing

The results of the analysis can be obtained using the results function and displayed to the screen. Each row is a particular gene measured in the study (i.e. all genes in the organism being studied) and each column reports some aspect of the differential expression analysis for that gene. Note that all genes are reported. At this stage the gene identifiers are not very informative, something we will fix in the next section.

results(de.cellType)
log2 fold change (MLE): CellType luminal vs basal 
Wald test p-value: CellType luminal vs basal 
DataFrame with 17973 rows and 6 columns
                           baseMean     log2FoldChange
                          <numeric>          <numeric>
ENSMUSG00000000001 4068.33959643508  0.385871582598132
ENSMUSG00000000028 203.536232806184  0.811809379599301
ENSMUSG00000000037 121.264614183815  -1.79043393209216
ENSMUSG00000000056 437.163852367364 0.0464783556140183
ENSMUSG00000000058 3648.16003486305   0.14655749676048
...                             ...                ...
ENSMUSG00000115744  6.5102254866026 -0.196252877440725
ENSMUSG00000115798 8.61324603359873   0.62806875352405
ENSMUSG00000115808 5.69983980001993  -1.87315498117461
ENSMUSG00000115812 3.48530425952456 0.0613115285646093
ENSMUSG00000115814  9.3358310615054 -0.646037485661208
                               lfcSE               stat
                           <numeric>          <numeric>
ENSMUSG00000000001 0.140534932730412   2.74573428186961
ENSMUSG00000000028 0.427713111143038   1.89802313384733
ENSMUSG00000000037 0.325457628706606  -5.50128119352274
ENSMUSG00000000056 0.336482286230572  0.138130170638966
ENSMUSG00000000058 0.498353539580032  0.294083386834144
...                              ...                ...
ENSMUSG00000115744 0.484587611080003 -0.404989465172943
ENSMUSG00000115798  0.52295422261736   1.20100140004721
ENSMUSG00000115808 0.678199955662751  -2.76195090479492
ENSMUSG00000115812 0.669781817316698 0.0915395535970767
ENSMUSG00000115814 0.606181994081262  -1.06574839234602
                                 pvalue                padj
                              <numeric>           <numeric>
ENSMUSG00000000001  0.00603756420772203  0.0133708069701353
ENSMUSG00000000028   0.0576930346502584  0.0969433024686542
ENSMUSG00000000037 3.77041320862499e-08 2.2514645684759e-07
ENSMUSG00000000056    0.890137541600907   0.919109240769746
ENSMUSG00000000058    0.768694185261808   0.826103590875375
...                                 ...                 ...
ENSMUSG00000115744    0.685485254686406   0.757823742959555
ENSMUSG00000115798     0.22975065824174   0.313957596604923
ENSMUSG00000115808  0.00574571175591682  0.0127875558117554
ENSMUSG00000115812    0.927063878924631   0.947700484484525
ENSMUSG00000115814    0.286537395618647   0.376541677087853

The output can be converted into a data frame and manipulated in the usual manner. It is recommended to use dplyr to manipulate the data frames with the standard set of operations detailed on the dplyr cheatsheet

  • select to pick which columns to display
  • filter to restrict the rows
  • mutate to add new variables to the data frame
  • arrange to order the data frame according to values of a column

However, dpylr does not like data frame that have rownames. We can use the rownames_to_column function from the tibble package to add an extra column that contains the Ensembl gene IDs.

The %>% symbol refers to the piping operation in R, which is a way of chaining operations together.

library(dplyr)
library(tibble)
results.cellType <- as.data.frame(results(de.cellType)) %>% 
  rownames_to_column("GeneID")  
results.cellType

We can sort the rows by adjusted p-value and then print the first 10 rows.

arrange(results.cellType, padj) %>%  
  head(n=10)

Or we can sort the rows and then write the resulting data frame to a file.

arrange(results.cellType, padj) %>%
  write.csv("basal_vs_luminal_DESeq_all.csv")
arrange(results.cellType, padj) %>%
  filter(padj < 0.05) %>% 
  write.csv("basal_vs_luminal_DESeq_DE.csv")

Challenge 1

  1. Re-run the analysis to find differentially-expressed genes between the developmental stages virgin and lactation
  2. Write a csv file that contains results for the genes that have a p-value less than 0.05 and a log2 fold change more than 1, or less than -1. HINT: So that we don’t overwrite our results so far, it may be convenient to create a new DESeqDataSet object for the new differential expression analysis.
dds.status <- dds
design(dds.status) <- ......

Changing the direction of the contrast

In this initial analyis DESeq2 has automatically decided which member of our sample groups to use as our baseline (basal in this case) so that the log2 fold changes are reported with a positve value meaning higher expression in luminal. If we want to change this behaviour we can change the contrast argument in the results function

## This should give the same as the table above
results(de.cellType, contrast=c("CellType","luminal","basal"))
## Changing the direction of the contrast
results(de.cellType, contrast=c("CellType","basal","luminal"))

If we change to performing differential expression analysis on the Status variable then there are various contrasts that can be made; pregnant vs lactation, lactation vs virgin etc. When the results function is run the table that is displayed is for the contrast virgin vs lactate. The resultsNames function can tell us which other contrasts we can access.

dds.status <- dds
design(dds.status) <- ~Status
de.status <- DESeq(dds.status)
using pre-existing normalization factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
resultsNames(de.status)
[1] "Intercept"                     "Status_pregnancy_vs_lactation"
[3] "Status_virgin_vs_lactation"   
results.status <- data.frame(results(de.status))

Intersecting gene lists

A venn diagram is a common way of visualising the overlap between two genelists. We need to create a data frame where each column indicates whether each gene is differentially expressed in a particular contrast or not. To create such columns we can do a logical test on the adjusted p-values from our results tables.

venn_data <- data.frame(CellType = results.cellType$padj<0.05,
                        Status = results.status$padj < 0.05)
library(limma)
vennDiagram(venn_data)

Challenge 2

  1. Use a venn diagram to visualise the overlap in the genes found to be differentially expressed in the pregnant vs virgin and lactation vs virgin contrasts.
  2. How many genes are in common?

Fitting alternative models to the data

DESEq2 allows for more complicated models to be fit to the data. For guidance on how to fit more complicated models you can consult the DESeq2 vignette, the limma user guide or the Bioconductor mailing list.

In particular, DESeq2 allows multi-factor models which can account for other sources of variation in the data such as batches or gender.

Lets suppose that we wanted the different between virgin and lactatin individuals, but controlling for CellType. The main assumption being that the effect of Status is the same regardless of CellType The design for such an analysis would be:-

dds.mf <- dds
design(dds.mf) <- ~CellType+Status
de.mf <- DESeq(dds.mf)
using pre-existing normalization factors
estimating dispersions
gene-wise dispersion estimates
mean-dispersion relationship
final dispersion estimates
fitting model and testing
results.mf <- results(de.mf,contrast=c("Status","lactation","virgin"))
results.mf
log2 fold change (MLE): Status lactation vs virgin 
Wald test p-value: Status lactation vs virgin 
DataFrame with 17973 rows and 6 columns
                           baseMean      log2FoldChange
                          <numeric>           <numeric>
ENSMUSG00000000001 4068.33959643508  -0.157073961462995
ENSMUSG00000000028 203.536232806184  -0.463768112351007
ENSMUSG00000000037 121.264614183815  -0.125203128964674
ENSMUSG00000000056 437.163852367364   0.322627299307911
ENSMUSG00000000058 3648.16003486305    1.61799767940002
...                             ...                 ...
ENSMUSG00000115744  6.5102254866026   -0.68542697373131
ENSMUSG00000115798 8.61324603359873   -1.12506788283054
ENSMUSG00000115808 5.69983980001993    1.46274904838166
ENSMUSG00000115812 3.48530425952456  -0.268958050014585
ENSMUSG00000115814  9.3358310615054 -0.0601959612877811
                               lfcSE                stat
                           <numeric>           <numeric>
ENSMUSG00000000001 0.112030929585792   -1.40205889609002
ENSMUSG00000000028 0.538243641584199  -0.861632310204371
ENSMUSG00000000037 0.408710289357286  -0.306337110234149
ENSMUSG00000000056  0.42532383259027   0.758545076919567
ENSMUSG00000000058 0.353421127866108    4.57810117116992
...                              ...                 ...
ENSMUSG00000115744 0.642620944270856   -1.06661163138563
ENSMUSG00000115798 0.600733305718197   -1.87282421687189
ENSMUSG00000115808 0.745592394541443    1.96186154672525
ENSMUSG00000115812 0.891160642964858  -0.301806472422045
ENSMUSG00000115814 0.719709561366585 -0.0836392407702365
                                 pvalue                 padj
                              <numeric>            <numeric>
ENSMUSG00000000001    0.160897660045662    0.298572057592648
ENSMUSG00000000028    0.388889885208587    0.554834325762911
ENSMUSG00000000037    0.759347986603781    0.852902660201997
ENSMUSG00000000056    0.448124738284556     0.60949480228342
ENSMUSG00000000058 4.69215756155008e-06 5.80255648055399e-05
...                                 ...                  ...
ENSMUSG00000115744    0.286147245969111    0.450595189259171
ENSMUSG00000115798   0.0610926627298804    0.145637759255059
ENSMUSG00000115808    0.049778605778987    0.124363730070539
ENSMUSG00000115812     0.76279959654206    0.855617282341187
ENSMUSG00000115814    0.933343266604123    0.962308323815519

Exporting normalized counts

The DESeq workflow applies median of ratios normalization that accounts for differences in sequencing depth between samples. The user does not usually need to run this step. However, if you want a matrix of counts for some application outside of Bioconductor the values can be extracted from the dds object.

dds <- estimateSizeFactors(dds) 
countMatrix <-counts(dds, normalized=TRUE)
head(countMatrix)
write.csv(countMatrix,file="normalized_counts.csv")
save(de.cellType,de.status,de.mf, file="Robjects/DE.Rdata")

Lun, Aaron T L, Yunshun Chen, and Gordon K Smyth. 2016. “It’s DE-licious: A Recipe for Differential Expression Analyses of RNA-seq Experiments Using Quasi-Likelihood Methods in edgeR.” Methods in Molecular Biology (Clifton, N.J.) 1418 (January): 391–416. doi:10.1007/978-1-4939-3578-9\_19.

LS0tDQp0aXRsZTogIlJOQS1zZXEgYW5hbHlzaXMgaW4gUiINCnN1YnRpdGxlOiAiRGlmZmVyZW50aWFsIEV4cHJlc3Npb24gb2YgUk5BLXNlcSBkYXRhIg0KYXV0aG9yOiAiTWFyayBEdW5uaW5nIg0KZGF0ZTogJ2ByIGZvcm1hdChTeXMudGltZSgpLCAiTGFzdCBtb2RpZmllZDogJWQgJWIgJVkiKWAnDQpvdXRwdXQ6DQogIGh0bWxfbm90ZWJvb2s6DQogICAgdG9jOiB5ZXMNCiAgICB0b2NfZmxvYXQ6IHllcw0KICBodG1sX2RvY3VtZW50Og0KICAgIHRvYzogeWVzDQogICAgdG9jX2Zsb2F0OiB5ZXMNCm1pbnV0ZXM6IDMwMA0KbGF5b3V0OiBwYWdlDQpiaWJsaW9ncmFwaHk6IHJlZi5iaWINCi0tLQ0KDQpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0NCmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSkNCmBgYA0KDQoqKk9yaWdpbmFsIEF1dGhvcnM6IEJlbGluZGEgUGhpcHNvbiwgQW5uYSBUcmlnb3MsIE1hdHQgUml0Y2hpZSwgTWFyaWEgRG95bGUsIEhhcnJpZXQgRGFzaG5vdywgQ2hhcml0eSBMYXcqKiwgKipTdGVwaGFuZSBCYWxsZXJlYXUsIE9zY2FyIFJ1ZWRhLCBBc2hsZXkgU2F3bGUqKg0KQmFzZWQgb24gdGhlIGNvdXJzZSBbUk5Bc2VxIGFuYWx5c2lzIGluIFJdKGh0dHA6Ly9jb21iaW5lLWF1c3RyYWxpYS5naXRodWIuaW8vMjAxNi0wNS0xMS1STkFzZXEvKSBkZWxpdmVyZWQgb24gTWF5IDExLzEydGggMjAxNiBhbmQgbW9kaWZpZWQgYnkgQ2FuY2VyIFJlc2VhcmNoIFVrIENhbWJyaWRnZSBDZW50cmUgZm9yIHRoZSBbRnVuY3Rpb25hbCBHZW5vbWljcyBBdXR1bW4gU2Nob29sIDIwMTddKGh0dHBzOi8vYmlvaW5mb3JtYXRpY3MtY29yZS1zaGFyZWQtdHJhaW5pbmcuZ2l0aHViLmlvL2NydWstYXV0dW1uLXNjaG9vbC0yMDE3LykNCg0KIyMgUmVzb3VyY2VzIGFuZCBkYXRhIGZpbGVzDQoNClRoaXMgbWF0ZXJpYWwgaGFzIGJlZW4gY3JlYXRlZCB1c2luZyB0aGUgZm9sbG93aW5nIHJlc291cmNlczogIA0KDQotIGh0dHA6Ly93d3cuc3RhdHNjaS5vcmcvc215dGgvcHVicy9RTGVkZ2VSUHJlcHJpbnQucGRmIFtATHVuMjAxNl0gIA0KLSBodHRwOi8vbW9uYXNoYmlvaW5mb3JtYXRpY3NwbGF0Zm9ybS5naXRodWIuaW8vUk5Bc2VxLURFLWFuYWx5c2lzLXdpdGgtUi85OS1STkFzZXFfREVfYW5hbHlzaXNfd2l0aF9SLmh0bWwgIA0KLSBodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwNCi0gaHR0cHM6Ly9iaW9jb25kdWN0b3IuZ2l0aHViLmlvL0Jpb2NXb3Jrc2hvcHMvcm5hLXNlcS1kYXRhLWFuYWx5c2lzLXdpdGgtZGVzZXEyLmh0bWwNCg0KDQoNCiMjIERpZmZlcmVudGlhbCBleHByZXNzaW9uIHdpdGggYERFU2VxMmANCg0KTm93IHRoYXQgd2UgYXJlIGhhcHB5IHRoYXQgd2UgaGF2ZSBub3JtYWxpc2VkIHRoZSBkYXRhIGFuZCB0aGF0IHRoZSBxdWFsaXR5IGxvb2tzIGdvb2QsIHdlIGNhbiBjb250aW51ZSB0byB0ZXN0aW5nIGZvciBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgZ2VuZXMuIFRoZXJlIGFyZSBhIG51bWJlciBvZiBwYWNrYWdlcyB0byBhbmFseXNlIFJOQS1TZXEgZGF0YS4gTW9zdCBwZW9wbGUgdXNlIGBERVNlcTJgIG9yIGBlZGdlUmAuIFdlIHdpbGwgdXNlIGBERVNlcTJgIGZvciB0aGUgcmVzdCBvZiB0aGlzIHByYWN0aWNhbC4NCg0KKipGaXJzdCBtYWtlIHN1cmUgd2UgaGF2ZSBhbGwgdGhlIG9iamVjdHMgYW5kIGxpYnJhcmllcyBsb2FkZWQqDQoNCmBgYHtyfQ0KbGlicmFyeShERVNlcTIpDQpsaWJyYXJ5KHR4aW1wb3J0KQ0KYGBgDQoNCg0KIyMjIFJlY2FwIG9mIHByZS1wcm9jZXNzaW5nDQoNClRoZSBwcmV2aW91cyBzZWN0aW9uIHdhbGtlZC10aHJvdWdoIHRoZSBwcmUtcHJvY2Vzc2luZyBhbmQgdHJhbnNmb3JtYXRpb24gb2YgdGhlIGNvdW50IGRhdGEuIEhlcmUsIGZvciBjb21wbGV0ZW5lc3MsIHdlIGxpc3QgdGhlIG1pbmltYWwgc3RlcHMgcmVxdWlyZWQgdG8gcHJvY2VzcyB0aGUgZGF0YSBwcmlvciB0byBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcy4NCg0KTm90ZSB0aGF0IGFsdGhvdWdoIHdlIHNwZW50IHNvbWUgdGltZSBsb29raW5nIGF0IHRoZSBxdWFsaXR5IG9mIG91ciBkYXRhICwgdGhlc2Ugc3RlcHMgYXJlIG5vdCByZXF1aXJlZCBwcmlvciB0byBwZXJmb3JtaW5nIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIHNvIGFyZSBub3Qgc2hvd24gaGVyZS4gUmVtZW1iZXIsIGBERVNlcTJgIFtyZXF1aXJlcyByYXcgY291bnRzXShodHRwOi8vYmlvY29uZHVjdG9yLm9yZy9wYWNrYWdlcy9kZXZlbC9iaW9jL3ZpZ25ldHRlcy9ERVNlcTIvaW5zdC9kb2MvREVTZXEyLmh0bWwjd2h5LXVuLW5vcm1hbGl6ZWQtY291bnRzKSBzbyB0aGUgYHZzdGAgdHJhbnNmb3JtYXRpb24gaXMgbm90IHNob3duIGFzIHBhcnQgb2YgdGhpcyBiYXNpYyBwcm90b2NvbC4NCg0KYGBge3IgZXZhbD1GQUxTRX0NCg0KZGlycyA8LSBsaXN0LmZpbGVzKGZpbGUucGF0aCgic2FsbW9uX3F1YW50LyIpKQ0KcXVhbnRfZmlsZXMgPC0gcGFzdGUwKCJzYWxtb25fcXVhbnQvIixkaXJzLCIvcXVhbnQuc2YuZ3oiKQ0KbmFtZXMocXVhbnRfZmlsZXMpIDwtIGRpcnMNCg0KdHgyZ2VuZSA8LSByZWFkLmNzdigidHgyZ2VuZS5jc3YiKQ0KDQp0eGkgPC0gdHhpbXBvcnQocXVhbnRfZmlsZXMsdHlwZT0ic2FsbW9uIix0eDJnZW5lID0gdHgyZ2VuZSxpZ25vcmVUeFZlcnNpb24gPSBUUlVFKQ0KDQpzYW1wbGVpbmZvIDwtIHJlYWQuZGVsaW0oIm1ldGFfZGF0YS9zYW1wbGVJbmZvX2NvcnJlY3RlZC50eHQiKQ0Kcm93bmFtZXMoc2FtcGxlaW5mbykgPC0gc2FtcGxlaW5mbyRydW4NCg0KZGRzIDwtIERFU2VxRGF0YVNldEZyb21UeGltcG9ydCh0eGksIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb2xEYXRhID0gc2FtcGxlaW5mbywNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGVzaWduIDwtIH5DZWxsVHlwZSkNCmtlZXAgPC0gcm93U3Vtcyhhc3NheShkZHMpID49IDUpID49IDINCmRkcyA8LSBkZHNba2VlcCxdDQpgYGANCg0KV2UgYWxzbyBoYXZlIHRoZSBvdXRwdXQgb2YgdGhlIHByZS1wcm9jZXNzaW5nIHNlY3Rpb24gc2F2ZWQgYXMgYW4gUiBvYmplY3QgaWYgeW91IGRpZG4ndCBtYW5hZ2UgdG8gY29tcGxldGUgdGhlc2Ugc3RlcHMuDQoNCmBgYHtyfQ0KIyMgT25seSBydW4gaWYgeW91IGRpZG4ndCBjb21wbGV0ZSB0aGUgcHJldmlvdXMgc2VjdGlvbiBvbiBwcmUtcHJvY2Vzc2luZw0KbG9hZCgiUm9iamVjdHMvcHJlcHJvY2Vzc2luZy5SZGF0YSIpDQpgYGANCg0KDQoNCiMjIERpZmZlcmVudGlhbCBFeHByZXNzaW9uIHdpdGggREVTZXEyDQoNCldlIGhhdmUgcHJldmlvdXNseSBkZWZpbmVkIHRoZSB0ZXN0IGNvbmRpdGlvbiB1c2luZyB0aGUgYGRlc2lnbmAgYXJndW1lbnQgd2hlbiB3ZSBjcmVhdGVkIHRoZSBvYmplY3QuIFRoaXMgY2FuIGJlIGNoZWNrZWQgdXNpbmcgdGhlIGBkZXNpZ25gIGZ1bmN0aW9uLg0KDQpUeXBpY2FsbHkgd2UgZGVjaWRlIHRoZSBkZXNpZ24gZm9yIHRoZSBhbmFseXNpcyB3aGVuIHdlIGNyZWF0ZSB0aGUgREVTZXEyIG9iamVjdHMsIGJ1dCBpdCBjYW4gYmUgbW9kaWZpZWQgcHJpb3IgdG8gdGhlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzDQoNCmBgYHtyfQ0KY29sRGF0YShkZHMpDQpkZXNpZ24oZGRzKSA8LSB+Q2VsbFR5cGUNCmBgYA0KDQoNCg0KVGhlIGZ1bmN0aW9uIHJ1bnMgYSBjb3VwbGUgb2YgcHJvY2Vzc2luZyBzdGVwcyBhdXRvbWF0aWNhbGx5IHRvIGFkanVzdCBmb3IgZGlmZmVyZW50IGxpYnJhcnkgc2l6ZSBhbmQgZ2VuZS13aXNlIHZhcmlhYmlsaXksIHdoaWNoIHlvdSBjYW4gcmVhZCBhYm91dCBpbiB0aGUgW0RFU2VxMiB2aWduZXR0ZV0oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sI3RoZS1kZXNlcTItbW9kZWwpLg0KDQpGaXJzdGx5IGFwcGx5IHRoZSBtZWRpYW4gcmF0aW8gbm9ybWFsaXNhdGlvbiBtZXRob2QgdG8gY29tcGVuc2F0ZSBmb3IgZGlmZmVyZW5jZXMgaW4gbGlicmFyeSBzaXplcw0KDQpgYGB7ciBldmFsPUZBTFNFfQ0KZGRzIDwtIGVzdGltYXRlU2l6ZUZhY3RvcnMoZGRzKQ0KDQpgYGANCg0KZXN0aW1hdGUgdGhlIGRpc3BlcnNpb24gZm9yIGVhY2ggZ2VuZQ0KDQpgYGB7ciBldmFsPUZBTFNFfQ0KZGRzIDwtIGVzdGltYXRlRGlzcGVyc2lvbnMoZGRzKQ0KYGBgDQoNCkFwcGx5IHN0YXRpc3RpY2FsIHRlc3RpbmcgYmFzZWQgb24gdGhlIG5lZ2F0aXZlIGJpbm9taWFsIGRpc3RyaWJ1dGlvbi4NCg0KYGBge3IgZXZhbD1GQUxTRX0NCmRkcyA8LSBuYmlub21XYWxkVGVzdChkZHMpDQpgYGANCg0KRm9ydHVuYXRlbHksIHRoZXJlIGlzIG9uZSBjb252ZW5pZW50IGZ1bmN0aW9uIHRoYXQgd2lsbCBhcHBseSB0aGUgdGhyZWUgc3RlcHMNCg0KYGBge3J9DQpkZS5jZWxsVHlwZSA8LSBERVNlcShkZHMpDQoNCmBgYA0KDQoNCg0KVGhlIHJlc3VsdHMgb2YgdGhlIGFuYWx5c2lzIGNhbiBiZSBvYnRhaW5lZCB1c2luZyB0aGUgYHJlc3VsdHNgIGZ1bmN0aW9uIGFuZCBkaXNwbGF5ZWQgdG8gdGhlIHNjcmVlbi4gRWFjaCByb3cgaXMgYSBwYXJ0aWN1bGFyIGdlbmUgbWVhc3VyZWQgaW4gdGhlIHN0dWR5IChpLmUuIGFsbCBnZW5lcyBpbiB0aGUgb3JnYW5pc20gYmVpbmcgc3R1ZGllZCkgYW5kIGVhY2ggY29sdW1uIHJlcG9ydHMgc29tZSBhc3BlY3Qgb2YgdGhlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIGFuYWx5c2lzIGZvciB0aGF0IGdlbmUuIE5vdGUgdGhhdCBhbGwgZ2VuZXMgYXJlIHJlcG9ydGVkLiBBdCB0aGlzIHN0YWdlIHRoZSBnZW5lIGlkZW50aWZpZXJzIGFyZSBub3QgdmVyeSBpbmZvcm1hdGl2ZSwgc29tZXRoaW5nIHdlIHdpbGwgZml4IGluIHRoZSBuZXh0IHNlY3Rpb24uIA0KDQpgYGB7cn0NCnJlc3VsdHMoZGUuY2VsbFR5cGUpDQpgYGANCg0KVGhlIG91dHB1dCBjYW4gYmUgY29udmVydGVkIGludG8gYSBkYXRhIGZyYW1lIGFuZCBtYW5pcHVsYXRlZCBpbiB0aGUgdXN1YWwgbWFubmVyLiBJdCBpcyByZWNvbW1lbmRlZCB0byB1c2UgYGRwbHlyYCB0byBtYW5pcHVsYXRlIHRoZSBkYXRhIGZyYW1lcyB3aXRoIHRoZSBzdGFuZGFyZCBzZXQgb2Ygb3BlcmF0aW9ucyBkZXRhaWxlZCBvbiB0aGUgW2RwbHlyIGNoZWF0c2hlZXRdKGh0dHBzOi8vd3d3LnJzdHVkaW8uY29tL3dwLWNvbnRlbnQvdXBsb2Fkcy8yMDE1LzAyL2RhdGEtd3JhbmdsaW5nLWNoZWF0c2hlZXQucGRmKQ0KDQotIGBzZWxlY3RgIHRvIHBpY2sgd2hpY2ggY29sdW1ucyB0byBkaXNwbGF5DQotIGBmaWx0ZXJgIHRvIHJlc3RyaWN0IHRoZSByb3dzDQotIGBtdXRhdGVgIHRvIGFkZCBuZXcgdmFyaWFibGVzIHRvIHRoZSBkYXRhIGZyYW1lDQotIGBhcnJhbmdlYCB0byBvcmRlciB0aGUgZGF0YSBmcmFtZSBhY2NvcmRpbmcgdG8gdmFsdWVzIG9mIGEgY29sdW1uDQoNCkhvd2V2ZXIsIGBkcHlscmAgZG9lcyBub3QgbGlrZSBkYXRhIGZyYW1lIHRoYXQgaGF2ZSByb3duYW1lcy4gV2UgY2FuIHVzZSB0aGUgYHJvd25hbWVzX3RvX2NvbHVtbmAgZnVuY3Rpb24gZnJvbSB0aGUgYHRpYmJsZWAgcGFja2FnZSB0byBhZGQgYW4gZXh0cmEgY29sdW1uIHRoYXQgY29udGFpbnMgdGhlIEVuc2VtYmwgZ2VuZSBJRHMuDQoNClRoZSBgJT4lYCBzeW1ib2wgcmVmZXJzIHRvIHRoZSBwaXBpbmcgb3BlcmF0aW9uIGluIFIsIHdoaWNoIGlzIGEgd2F5IG9mIGNoYWluaW5nIG9wZXJhdGlvbnMgdG9nZXRoZXIuIA0KDQpgYGB7cn0NCmxpYnJhcnkoZHBseXIpDQpsaWJyYXJ5KHRpYmJsZSkNCnJlc3VsdHMuY2VsbFR5cGUgPC0gYXMuZGF0YS5mcmFtZShyZXN1bHRzKGRlLmNlbGxUeXBlKSkgJT4lIA0KICByb3duYW1lc190b19jb2x1bW4oIkdlbmVJRCIpICANCg0KcmVzdWx0cy5jZWxsVHlwZQ0KYGBgDQoNCldlIGNhbiBzb3J0IHRoZSByb3dzIGJ5IGFkanVzdGVkIHAtdmFsdWUgYW5kIHRoZW4gcHJpbnQgdGhlIGZpcnN0IDEwIHJvd3MuDQoNCmBgYHtyfQ0KYXJyYW5nZShyZXN1bHRzLmNlbGxUeXBlLCBwYWRqKSAlPiUgIA0KICBoZWFkKG49MTApDQpgYGANCg0KT3Igd2UgY2FuIHNvcnQgdGhlIHJvd3MgYW5kIHRoZW4gd3JpdGUgdGhlIHJlc3VsdGluZyBkYXRhIGZyYW1lIHRvIGEgZmlsZS4NCg0KYGBge3J9DQphcnJhbmdlKHJlc3VsdHMuY2VsbFR5cGUsIHBhZGopICU+JQ0KICB3cml0ZS5jc3YoImJhc2FsX3ZzX2x1bWluYWxfREVTZXFfYWxsLmNzdiIpDQpgYGANCg0KDQpgYGB7cn0NCmFycmFuZ2UocmVzdWx0cy5jZWxsVHlwZSwgcGFkaikgJT4lDQogIGZpbHRlcihwYWRqIDwgMC4wNSkgJT4lIA0KICB3cml0ZS5jc3YoImJhc2FsX3ZzX2x1bWluYWxfREVTZXFfREUuY3N2IikNCmBgYA0KDQoNCj4gIyMgQ2hhbGxlbmdlIDEgey5jaGFsbGVuZ2V9DQo+DQo+IDEuIFJlLXJ1biB0aGUgYW5hbHlzaXMgdG8gZmluZCBkaWZmZXJlbnRpYWxseS1leHByZXNzZWQgZ2VuZXMgYmV0d2VlbiB0aGUgZGV2ZWxvcG1lbnRhbCBzdGFnZXMgKnZpcmdpbiogYW5kICpsYWN0YXRpb24qDQo+IDIuIFdyaXRlIGEgY3N2IGZpbGUgdGhhdCBjb250YWlucyByZXN1bHRzIGZvciB0aGUgZ2VuZXMgdGhhdCBoYXZlIGEgcC12YWx1ZSBsZXNzIHRoYW4gMC4wNSBhbmQgYSBsb2cyIGZvbGQgY2hhbmdlIG1vcmUgdGhhbiAxLCBvciBsZXNzIHRoYW4gLTEuDQo+IEhJTlQ6IFNvIHRoYXQgd2UgZG9uJ3Qgb3ZlcndyaXRlIG91ciByZXN1bHRzIHNvIGZhciwgaXQgbWF5IGJlIGNvbnZlbmllbnQgdG8gY3JlYXRlIGEgbmV3IGBERVNlcURhdGFTZXRgIG9iamVjdCBmb3IgdGhlIG5ldyBkaWZmZXJlbnRpYWwgZXhwcmVzc2lvbiBhbmFseXNpcy4NCg0KYGBge3IgZXZhbD1GQUxTRX0NCmRkcy5zdGF0dXMgPC0gZGRzDQpkZXNpZ24oZGRzLnN0YXR1cykgPC0gLi4uLi4uDQpgYGANCg0KIyMjIENoYW5naW5nIHRoZSBkaXJlY3Rpb24gb2YgdGhlIGNvbnRyYXN0DQoNCkluIHRoaXMgaW5pdGlhbCBhbmFseWlzIGBERVNlcTJgIGhhcyBhdXRvbWF0aWNhbGx5IGRlY2lkZWQgd2hpY2ggbWVtYmVyIG9mIG91ciBzYW1wbGUgZ3JvdXBzIHRvIHVzZSBhcyBvdXIgYmFzZWxpbmUgKGBiYXNhbGAgaW4gdGhpcyBjYXNlKSBzbyB0aGF0IHRoZSBsb2cyIGZvbGQgY2hhbmdlcyBhcmUgcmVwb3J0ZWQgd2l0aCBhIHBvc2l0dmUgdmFsdWUgbWVhbmluZyBoaWdoZXIgZXhwcmVzc2lvbiBpbiBgbHVtaW5hbGAuIElmIHdlIHdhbnQgdG8gY2hhbmdlIHRoaXMgYmVoYXZpb3VyIHdlIGNhbiBjaGFuZ2UgdGhlIGBjb250cmFzdGAgYXJndW1lbnQgaW4gdGhlIGByZXN1bHRzYCBmdW5jdGlvbg0KDQoNCmBgYHtyIGV2YWw9RkFMU0V9DQojIyBUaGlzIHNob3VsZCBnaXZlIHRoZSBzYW1lIGFzIHRoZSB0YWJsZSBhYm92ZQ0KcmVzdWx0cyhkZS5jZWxsVHlwZSwgY29udHJhc3Q9YygiQ2VsbFR5cGUiLCJsdW1pbmFsIiwiYmFzYWwiKSkNCiMjIENoYW5naW5nIHRoZSBkaXJlY3Rpb24gb2YgdGhlIGNvbnRyYXN0DQpyZXN1bHRzKGRlLmNlbGxUeXBlLCBjb250cmFzdD1jKCJDZWxsVHlwZSIsImJhc2FsIiwibHVtaW5hbCIpKQ0KDQpgYGANCg0KSWYgd2UgY2hhbmdlIHRvIHBlcmZvcm1pbmcgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gYW5hbHlzaXMgb24gdGhlIGBTdGF0dXNgIHZhcmlhYmxlIHRoZW4gdGhlcmUgYXJlIHZhcmlvdXMgY29udHJhc3RzIHRoYXQgY2FuIGJlIG1hZGU7IGBwcmVnbmFudGAgdnMgYGxhY3RhdGlvbmAsIGBsYWN0YXRpb25gIHZzIGB2aXJnaW5gIGV0Yy4gV2hlbiB0aGUgYHJlc3VsdHNgIGZ1bmN0aW9uIGlzIHJ1biB0aGUgdGFibGUgdGhhdCBpcyBkaXNwbGF5ZWQgaXMgZm9yIHRoZSBjb250cmFzdCBgdmlyZ2luIHZzIGxhY3RhdGVgLiBUaGUgYHJlc3VsdHNOYW1lc2AgZnVuY3Rpb24gY2FuIHRlbGwgdXMgd2hpY2ggb3RoZXIgY29udHJhc3RzIHdlIGNhbiBhY2Nlc3MuDQoNCg0KYGBge3J9DQpkZHMuc3RhdHVzIDwtIGRkcw0KZGVzaWduKGRkcy5zdGF0dXMpIDwtIH5TdGF0dXMNCmRlLnN0YXR1cyA8LSBERVNlcShkZHMuc3RhdHVzKQ0KcmVzdWx0c05hbWVzKGRlLnN0YXR1cykNCnJlc3VsdHMuc3RhdHVzIDwtIGRhdGEuZnJhbWUocmVzdWx0cyhkZS5zdGF0dXMpKQ0KYGBgDQoNCiMjIyBJbnRlcnNlY3RpbmcgZ2VuZSBsaXN0cw0KDQpBIHZlbm4gZGlhZ3JhbSBpcyBhIGNvbW1vbiB3YXkgb2YgdmlzdWFsaXNpbmcgdGhlIG92ZXJsYXAgYmV0d2VlbiB0d28gZ2VuZWxpc3RzLiBXZSBuZWVkIHRvIGNyZWF0ZSBhIGRhdGEgZnJhbWUgd2hlcmUgZWFjaCBjb2x1bW4gaW5kaWNhdGVzIHdoZXRoZXIgZWFjaCBnZW5lIGlzIGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBpbiBhIHBhcnRpY3VsYXIgY29udHJhc3Qgb3Igbm90LiBUbyBjcmVhdGUgc3VjaCBjb2x1bW5zIHdlIGNhbiBkbyBhIGxvZ2ljYWwgdGVzdCBvbiB0aGUgYWRqdXN0ZWQgcC12YWx1ZXMgZnJvbSBvdXIgcmVzdWx0cyB0YWJsZXMuDQoNCg0KYGBge3IgbWVzc2FnZT1GQUxTRX0NCnZlbm5fZGF0YSA8LSBkYXRhLmZyYW1lKENlbGxUeXBlID0gcmVzdWx0cy5jZWxsVHlwZSRwYWRqPDAuMDUsDQogICAgICAgICAgICAgICAgICAgICAgICBTdGF0dXMgPSByZXN1bHRzLnN0YXR1cyRwYWRqIDwgMC4wNSkNCmxpYnJhcnkobGltbWEpDQp2ZW5uRGlhZ3JhbSh2ZW5uX2RhdGEpDQpgYGANCg0KDQo+ICMjIENoYWxsZW5nZSAyIHsuY2hhbGxlbmdlfQ0KPg0KPiAxLiBVc2UgYSB2ZW5uIGRpYWdyYW0gdG8gdmlzdWFsaXNlIHRoZSBvdmVybGFwIGluIHRoZSBnZW5lcyBmb3VuZCB0byBiZSBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgaW4gdGhlIGBwcmVnbmFudCB2cyB2aXJnaW5gIGFuZCBgIGxhY3RhdGlvbiB2cyB2aXJnaW5gIGNvbnRyYXN0cy4gDQo+IDIuIEhvdyBtYW55IGdlbmVzIGFyZSBpbiBjb21tb24/DQoNCg0KDQojIyMgRml0dGluZyBhbHRlcm5hdGl2ZSBtb2RlbHMgdG8gdGhlIGRhdGENCg0KYERFU0VxMmAgYWxsb3dzIGZvciBtb3JlIGNvbXBsaWNhdGVkIG1vZGVscyB0byBiZSBmaXQgdG8gdGhlIGRhdGEuIEZvciBndWlkYW5jZSBvbiBob3cgdG8gZml0IG1vcmUgY29tcGxpY2F0ZWQgbW9kZWxzIHlvdSBjYW4gY29uc3VsdCB0aGUgW0RFU2VxMiB2aWduZXR0ZV0oaHR0cDovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvZGV2ZWwvYmlvYy92aWduZXR0ZXMvREVTZXEyL2luc3QvZG9jL0RFU2VxMi5odG1sKSwgdGhlIFtsaW1tYSB1c2VyIGd1aWRlXShodHRwczovL3d3dy5iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL2xpbW1hL2luc3QvZG9jL3VzZXJzZ3VpZGUucGRmKSBvciB0aGUgQmlvY29uZHVjdG9yIG1haWxpbmcgbGlzdC4gDQoNCkluIHBhcnRpY3VsYXIsIERFU2VxMiBhbGxvd3MgW211bHRpLWZhY3RvciBtb2RlbHNdKGh0dHA6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzL2RldmVsL2Jpb2MvdmlnbmV0dGVzL0RFU2VxMi9pbnN0L2RvYy9ERVNlcTIuaHRtbCNtdWx0aS1mYWN0b3ItZGVzaWducykgd2hpY2ggY2FuIGFjY291bnQgZm9yIG90aGVyIHNvdXJjZXMgb2YgdmFyaWF0aW9uIGluIHRoZSBkYXRhIHN1Y2ggYXMgYmF0Y2hlcyBvciBnZW5kZXIuDQoNCkxldHMgc3VwcG9zZSB0aGF0IHdlIHdhbnRlZCB0aGUgZGlmZmVyZW50IGJldHdlZW4gdmlyZ2luIGFuZCBsYWN0YXRpbiBpbmRpdmlkdWFscywgYnV0IGNvbnRyb2xsaW5nIGZvciBgQ2VsbFR5cGVgLiBUaGUgbWFpbiBhc3N1bXB0aW9uIGJlaW5nIHRoYXQgdGhlIGVmZmVjdCBvZiBgU3RhdHVzYCBpcyB0aGUgc2FtZSByZWdhcmRsZXNzIG9mIGBDZWxsVHlwZWAgVGhlIGRlc2lnbiBmb3Igc3VjaCBhbiBhbmFseXNpcyB3b3VsZCBiZTotDQoNCmBgYHtyfQ0KZGRzLm1mIDwtIGRkcw0KZGVzaWduKGRkcy5tZikgPC0gfkNlbGxUeXBlK1N0YXR1cw0KZGUubWYgPC0gREVTZXEoZGRzLm1mKQ0KcmVzdWx0cy5tZiA8LSByZXN1bHRzKGRlLm1mLGNvbnRyYXN0PWMoIlN0YXR1cyIsImxhY3RhdGlvbiIsInZpcmdpbiIpKQ0KcmVzdWx0cy5tZg0KYGBgDQoNCiMjIyBFeHBvcnRpbmcgbm9ybWFsaXplZCBjb3VudHMNCg0KVGhlIGBERVNlcWAgd29ya2Zsb3cgYXBwbGllcyAqbWVkaWFuIG9mIHJhdGlvcyBub3JtYWxpemF0aW9uKiB0aGF0IGFjY291bnRzIGZvciBkaWZmZXJlbmNlcyBpbiBzZXF1ZW5jaW5nIGRlcHRoIGJldHdlZW4gc2FtcGxlcy4gVGhlIHVzZXIgZG9lcyBub3QgdXN1YWxseSBuZWVkIHRvIHJ1biB0aGlzIHN0ZXAuIEhvd2V2ZXIsIGlmIHlvdSB3YW50IGEgbWF0cml4IG9mIGNvdW50cyBmb3Igc29tZSBhcHBsaWNhdGlvbiBvdXRzaWRlIG9mIEJpb2NvbmR1Y3RvciB0aGUgdmFsdWVzIGNhbiBiZSBleHRyYWN0ZWQgZnJvbSB0aGUgYGRkc2Agb2JqZWN0Lg0KDQpgYGB7cn0NCmRkcyA8LSBlc3RpbWF0ZVNpemVGYWN0b3JzKGRkcykgDQpjb3VudE1hdHJpeCA8LWNvdW50cyhkZHMsIG5vcm1hbGl6ZWQ9VFJVRSkNCmhlYWQoY291bnRNYXRyaXgpDQp3cml0ZS5jc3YoY291bnRNYXRyaXgsZmlsZT0ibm9ybWFsaXplZF9jb3VudHMuY3N2IikNCmBgYA0KDQoNCmBgYHtyIGV2YWw9RkFMU0V9DQpzYXZlKGRlLmNlbGxUeXBlLGRlLnN0YXR1cyxkZS5tZiwgZmlsZT0iUm9iamVjdHMvREUuUmRhdGEiKQ0KYGBgDQoNCg==