About

Team

Current

Yongwook Choi
Agnes Chan

Former

Greg Sims
Jason Miller
Sean Murphy

This work is funded by National Institutes of Health grant number 5R01HG004701-03.

References

Choi Y, Sims GE, Murphy S, Miller JR, Chan AP (2012) Predicting the Functional Effect of Amino Acid Substitutions and Indels. PLoS ONE 7(10): e46688.
Choi Y (2012) A Fast Computation of Pairwise Sequence Alignment Scores Between a Protein and a Set of Single-Locus Variants of Another Protein. In Proceedings of the ACM Conference on Bioinformatics, Computational Biology and Biomedicine (BCB '12). ACM, New York, NY, USA, 414-417.

Overview - PROVEAN (Protein Variation Effect Analyzer) v1.1

PROVEAN Score
PROVEAN Peformance
Comparison with Other Tools for Single Amino Acid Substitutions

The results below are based on PROVEAN v1.1. For the performance of previous version (v1.0), please refer to this page.

1. PROVEAN Score

PROVEAN was developed to predict whether a protein sequence variation affects protein function.

PROVEAN is able to provide predictions for any type of protein sequence variations including the following.

Single or multiple amino acid substitutions
Single or multiple amino acid insertions
Single or multiple amino acid deletions

(a) Overview

An overview of the PROVEAN procedure is shown in Figure 1. Briefly, clustering of BLAST hits is performed by CD-HIT with a parameter of 75% global sequence identity. The top 30 clusters of closely related sequences form the supporting sequence set, which will be used to generate the prediction.

A delta alignment score is computed for each supporting sequence. The scores are then averaged within and across clusters to generate the final PROVEAN score. If the PROVEAN score is equal to or below a predefined threshold (e.g. -2.5), the protein variant is predicted to have a "deleterious" effect. If the PROVEAN score is above the threshold, the variant is predicted to have a "neutral" effect.

Figure 1. Computing the PROVEAN score.

(b) Setting the PROVEAN score threshold

PROVEAN introduces a delta alignment score based on the reference and variant versions of a protein query sequence with respect to sequence homologs collected from the NCBI NR protein database through BLAST. The PROVEAN score distribution of a set of 58K UniProt human protein variants with known functional outcome is shown in Figure 2.

For maximum separation of the deleterious and neutral variants for all 4 classes of human protein variants, the default score threshold is currently set at -2.5 for binary classification (i.e. deleterious vs neutral).

Figure 2. PROVEAN (v1.1) score distribution for 58K deleterious and neutral UniProt human protein variations.

PROVEAN score distribution

(c) How to increase stringency of detection

To increase the specificity of detection (i.e. find deleterious variants with higher confidence), a lower score threshold can also be used (e.g. -4.1). To increase the sensitivity of detection (i.e. find more deleterious variants by including those with lower confidence), a higher score threshold can also be used (e.g. -1.3). A summary of specificity and sensitivity obtained using different score cutoffs is shown in Figure 3.

Figure 3. Sensitivity and specificity at different PROVEAN score cutoffs.

(d) PROVEAN v1.1 compared to the previous version (v1.0)

In order to refine the supporting sequence set, some conditions were added and parameters were changed accordingly in the process of selecting supporting sequences.

In order to be used as the supporting sequence, a sequence must be at least 30% of the length of query sequence.
While clustering, each sequence in a cluster must be at least 30% of the length of its representative sequence.
With 75% global sequence identity, top 30 clusters were selected.

2. PROVEAN Performance

The overall accuracy for binary classification of protein variants (deleterious or neutral, classification property is being deleterious):

79.5% for UniProt human protein variations (download)
77.9% for UniProt non-human protein variations (other mammals, plants, bacteria, etc.) (download)

Table 1. Number of human protein variations and prediction accuracy of PROVEAN v1.1.

Variant types	Functional outcome			Peformance(%)
Variant types	Deleterious	Neutral	Total	Sensitivity	Specificity	Accuracy*	Balanced accuracy**
Single amino acid substitutions	20821	36825	57646	78.85	79.55	79.29	79.20
Deletions	652	77	729	96.78	66.23	93.55	81.51
Insertions	110	61	171	91.82	78.69	87.13	85.25
Replacements	79	59	138	93.67	64.41	81.16	79.04
Total	21662	37022	58684	79.51	79.49	79.50	79.50

Table 2. Number of non-human protein variations and prediction accuracy of PROVEAN v1.1.

Variant types	Functional outcome			Performance (%)
Variant types	Deleterious	Neutral	Total	Sensitivity	Specificity	Accuracy*	Balanced accuracy**
Single amino acid substitutions	14117	16498	30615	80.33	76.00	78.00	78.17
Deletions	142	227	369	82.39	61.23	69.38	71.81
Insertions	34	137	171	76.47	75.18	75.44	75.83
Replacements	886	1029	1915	86.79	63.46	74.26	75.13
Total	15179	17891	33070	80.72	75.09	77.67	77.90

Note:

*Accuracy=(TP + TN)/(TP + TN + FP + FN); affected by potential unequal sizes of deleterious and neutral datasets
**Balanced Acurracy=(Sn + Sp)/2; unaffected by potential unequal sizes of deleterious and neutral datasets
A score threshold of -2.583 was used.

TP: true positive; TN: true negative; FP: false positive; FN: false negative; Sn: sensitivity; Sp: specificity

3. Comparison with Other Tools for Single Amino Acid Substitutions

(a) Prediction accuracy of different tools

The performance of PROVEAN was compared with other prediction tools: Mutation Assessor, SIFT, PolyPhen-2, and Condel, using default score thresholds suggested by individual tools. Overall, the performance of PROVEAN is comparable to other tools as shown in Tables 3 and 4.

Table 3. Prediction accuracy of PROVEAN, Mutation Assessor, SIFT, PolyPhen-2, and Condel for the UniProt human protein variant datasets.

Prediction tools	Score thresholds	Human dataset					References
Prediction tools	Score thresholds	Sensitivity	Specificity	Accuracy*	Balanced accuracy**	No prediction	References
PROVEAN	-4.100 (default) -2.500 -1.300	56.17 79.76 90.28	90.34 78.63 61.65	78.00 79.04 71.99	73.25 79.19 75.96	0	Choi et al., 2012; web
Mutation Assessor	0.800 1.90	96.54 85.29	40.59 71.02	60.90 76.20	68.57 78.15	317 (0.55%)	Reva et al., 2011; web
SIFT	0.050	85.03	68.95	74.77	76.99	1147 (1.99%)	Kumar et al., 2009; web
PolyPhen-2	0.432	88.68	62.45	72.00	75.56	2279 (3.95%)	Adzhubei et al., 2010; web
Condel web server	0.469	93.84	46.23	64.67	70.04	7194 (12.48%)	González-Pérez and López-Bigas, 2011; web

Table 4. Prediction accuracy of PROVEAN, Mutation Assessor, SIFT, and PolyPhen-2 for the UniProt non-human protein variant datasets.

Prediction tools	Score thresholds	Non-human dataset
Prediction tools	Score thresholds	Sensitivity	Specificity	Accuracy*	Balanced accuracy**	No prediction
PROVEAN	-4.100 (default) -2.500 -1.300	59.76 81.11 89.39	86.67 75.22 61.63	74.26 77.94 74.43	73.22 78.17 75.51	0
Mutation Assessor	0.800 1.90	93.17 81.30	45.13 67.16	67.07 73.62	69.15 74.23	732 (2.39%)
SIFT	0.050	87.47	69.27	77.88	78.37	1542 (5.04%)
PolyPhen-2	0.432	87.77	65.81	76.11	76.79	1499 (4.90%)

Note:

The UniProt human and non-human protein variant datasets were used.
The comparison was performed only for single amino acid substitutions because other tools do not handle other types of variation such as amino acid insertion or deletion.
The "No prediction" column shows the number of variants for which the corresponding tool fails to provide a prediction.
PROVEAN v1.1, Mutation Assessor web server v1.0, SIFT v4.0.3, PolyPhen-2 webserver v2.1.0, and Condel web server v1.4 were used.
*Accuracy=(TP + TN)/(TP + TN + FP + FN); affected by potential unequal sizes of deleterious and neutral datasets
**Balanced Acurracy=(Sn + Sp)/2; unaffected by potential unequal sizes of deleterious and neutral datasets

TP: true positive; TN: true negative; FP: false positive; FN: false negative; Sn: sensitivity; Sp: specificity

(b) Prediction consistency among different tools

The prediction results for the UniProt human protein variant dataset were obtained from multiple tools including PROVEAN, SIFT, and PolyPhen-2, and are summarized in Venn diagrams shown in Figure 4. Overall, the prediction results for 78.5% (15,618/19,898) of disease-associated variants and 46.8% (16,244/34,701) common variants are in agreement and shared by all three tools. In addition, each tool produces correct predictions for distinct subsets of disease or neutral variants. Combining prediction results from multiple tools can increase the chance of identifying functional variants that had been missed by other tools.

Figure 4. Venn diagrams showing predictions from PROVEAN, SIFT, and PolyPhen-2 for the UniProt human protein variant dataset (score thresholds used: PROVEAN, -1.3; SIFT, 0.05; PolyPhen-2, 0.432).

Note:

The number and percentage of correctly predicted variants are shown next to each tool.