Skip to content

Commit

Permalink
Merge pull request #15 from turboflakes/track-bitfields
Browse files Browse the repository at this point in the history 
Track bitfields
  • Loading branch information
@paulormart
paulormart authored Jan 10, 2025
2 parents 8c5924c + 439f576 commit 9a702e9
Show file tree
Hide file tree
Showing 10 changed files with 774 additions and 260 deletions.
52 changes: 29 additions & 23 deletions LEGENDS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,22 @@ By typing `!legends` in one of the main public channels the following message wi
❒: Total number of authored blocks by the validator.
✓i: Total number of implicit votes by the validator.
✓e: Total number of explicit votes by the validator.
✗: Total number of missed votes by the validator.
MVR: Missed Votes Ratio (MVR) `MVR = (✗) / (✓i + ✓e + ✗)`.

GRD: Grade reflects the Backing Votes Ratio (BVR) `BVR = 1 - MVR` by the validator:
‣ A+ = BVR > 99%
‣ A = BVR > 95%
‣ B+ = BVR > 90%
‣ B = BVR > 80%
‣ C+ = BVR > 70%
‣ C = BVR > 60%
‣ D+ = BVR > 50%
‣ D = BVR > 40%
‣ F = BVR <= 40%
✗v: Total number of missed votes by the validator.
✓ba: Total number of blocks containing populated bitfields from the validator.
✗bu: Total number of blocks with bitfields unavailable or empty from the validator.
MVR: Missed Votes Ratio (MVR) `MVR = (✗) / (✓i + ✓e + ✗)`.
BAR: Bitfields Availability Ratio `(BAR = (✓ba) / (✓ba + ✗bu))`.

GRD: Grade is calculated as 75% of the Backing Votes Ratio (BVR = 1-MVR) combined with 25% of the Bitfields Availability Ratio (BAR) by one or more validators `(RATIO = BVR*0.75 + BAR*0.25)`:
‣ A+ = RATIO > 99%
‣ A = RATIO > 95%
‣ B+ = RATIO > 90%
‣ B = RATIO > 80%
‣ C+ = RATIO > 70%
‣ C = RATIO > 60%
‣ D+ = RATIO > 50%
‣ D = RATIO > 40%
‣ F = RATIO <= 40%

PPTS: Sum of para-validator points the validator earned.
TPTS: Sum of para-validator points + authored blocks points the validator earned.
Expand All @@ -39,9 +42,12 @@ A, B, C, D: Represents each validator in the same val. group as the subscribed v
❒: Total number of authored blocks.
✓i: Total number of implicit votes.
✓e: Total number of explicit votes.
✗: Total number of missed votes by the validator.
✗v: Total number of missed votes by the validator.
✓ba: Total number of blocks containing populated bitfields from the validator.
✗bu: Total number of blocks with bitfields unavailable or empty from the validator.
GRD: Grade reflects the Backing Votes Ratio.
MVR: Missed Votes Ratio.
MVR: Missed Votes Ratio.
BAR: Bitfields Availability Ratio.
PPTS: Sum of para-validator points the validator earned.
TPTS: Sum of para-validator points + authored blocks points the validator earned.
_Note: Val. groups and validators are sorted by para-validator points in descending order._
Expand All @@ -54,7 +60,7 @@ _Note: Val. groups and validators are sorted by para-validator points in descend
❒: Total number of authored blocks from all validators when assigned to the parachain.
✓i: Total number of implicit votes from all validators when assigned to the parachain.
✓e: Total number of explicit votes from all validators when assigned to the parachain.
✗: Total number of missed votes from all validators when assigned to the parachain.
✗v: Total number of missed votes from all validators when assigned to the parachain.
PPTS: Sum of para-validator points from all validators.
TPTS: Sum of para-validator points + authored blocks points from all validators.
_Note: Parachains are sorted by para-validator points in descending order._
Expand All @@ -63,16 +69,16 @@ _Note: Parachains are sorted by para-validator points in descending order._

`!subscribe insights`

Score: `score = (1 - mvr) * 0.75 + ((avg_pts - min_avg_pts) / (max_avg_pts - min_avg_pts)) * 0.18 + (pv_sessions / total_sessions) * 0.07`
Score: `score = (1 - mvr) * 0.55 + bar * 0.25 + ((avg_pts - min_avg_pts) / (max_avg_pts - min_avg_pts)) * 0.18 + (pv_sessions / total_sessions) * 0.07`
Commission Score: `commission_score = score * 0.25 + (1 - commission) * 0.75`

Timeline: Graphic performance representation in the last X sessions:
‣ ❚ = BVR >= 90%
‣ ❙ = BVR >= 60%
‣ ❘ = BVR >= 40%
‣ ! = BVR >= 20%
‣ ¿ = BVR < 20%
‣ ? = No-votes
‣ ❚ = RATIO >= 90%
‣ ❙ = RATIO >= 60%
‣ ❘ = RATIO >= 40%
‣ ! = RATIO >= 20%
‣ ¿ = RATIO < 20%
‣ ? = No-show
‣ • = Not P/V
‣ _ = Waiting

Expand Down
2 changes: 1 addition & 1 deletion SCORES.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Performance Score

`performance_score = (1 - mvr) * 0.75 + ((avg_pts - min_avg_pts) / (max_avg_pts - min_avg_pts)) * 0.18 + (pv_sessions / total_sessions) * 0.07`
`performance_score = (1 - mvr) * 0.50 + bar * 0.25 + ((avg_pts - min_avg_pts) / (max_avg_pts - min_avg_pts)) * 0.18 + (pv_sessions / total_sessions) * 0.07`

## Commission Score

Expand Down
144 changes: 105 additions & 39 deletions src/api/handlers/validators.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ use crate::config::CONFIG;
use crate::dn::try_fetch_stashes_from_remote_url;
use crate::errors::{ApiError, CacheError};
use crate::pools::{PoolId, PoolNominees};
use crate::records::{grade, EpochIndex, Grade};
use crate::records::{grade, BitfieldsRecord, EpochIndex, Grade};
use crate::report::Subset;
use actix_web::{
web::{Data, Json, Path, Query},
Expand Down Expand Up @@ -676,10 +676,10 @@ pub async fn get_validators(
// warn!("__serialized: {:?}", serialized);

//
// NOTE: the score is based on 5 key values, which will be aggregated in the following map tupple.
// NOTE: the tupple has a subset, 5 counters plus the final score like: (subset, para_epochs, para_points, explicit_votes, implicit_votes, missed_vote, score)
// NOTE: the score is based on 7 key values, which will be aggregated in the following map tupple.
// NOTE: the tuple has a subset, 7 counters plus the final score like: (subset, para_epochs, para_points, explicit_votes, implicit_votes, missed_votes, bitfields_availability, bitfields_unavailability, score)
//
let mut aggregator: BTreeMap<String, (Subset, u32, u32, u32, u32, u32, u32)> =
let mut aggregator: BTreeMap<String, (Subset, u32, u32, u32, u32, u32, u32, u32, u32)> =
BTreeMap::new();
let mut validators: BTreeMap<String, ValidatorResult> = BTreeMap::new();
let mut total_epochs: u32 = 0;
Expand All @@ -705,18 +705,29 @@ pub async fn get_validators(
cache.clone(),
)
.await?;
// NOTE: the tupple has a subset, 5 counters plus the final score like: (subset, para_epochs, para_points, explicit_votes, implicit_votes, missed_vote, score)
// NOTE: the tupple has a subset, 5 counters plus the final score like: (subset, para_epochs, para_points, explicit_votes, implicit_votes, missed_vote, bitfields_availability, bitfields_unavailability, score)
aggregator
.entry(val.address.clone())
.and_modify(|(subset, para_epochs, para_points, ev, iv, mv, _)| {
*subset = val.profile.subset.clone();
*para_epochs += 1;
*para_points += val.auth.para_points();
*ev += val.para_summary.explicit_votes();
*iv += val.para_summary.implicit_votes();
*mv += val.para_summary.missed_votes();
})
.or_insert((Subset::NotDefined, 0, 0, 0, 0, 0, 0));
.and_modify(
|(subset, para_epochs, para_points, ev, iv, mv, ba, bu, _)| {
*subset = val.profile.subset.clone();
*para_epochs += 1;
*para_points += val.auth.para_points();
*ev += val.para_summary.explicit_votes();
*iv += val.para_summary.implicit_votes();
*mv += val.para_summary.missed_votes();

if let Some(value) = val.para.get("bitfields") {
if let Ok(bitfields) =
serde_json::from_value::<BitfieldsRecord>(value.clone())
{
*ba += bitfields.availability();
*bu += bitfields.unavailability();
}
}
},
)
.or_insert((Subset::NotDefined, 0, 0, 0, 0, 0, 0, 0, 0));
validators.insert(val.address.clone(), val);
}
total_epochs += 1;
Expand All @@ -730,8 +741,10 @@ pub async fn get_validators(
// Normalize avg_para_points
let avg_para_points: Vec<u32> = aggregator_vec
.iter()
.filter(|(_, (_, para_epochs, _, _, _, _, _))| *para_epochs >= 1)
.map(|(_, (_, para_epochs, para_points, _, _, _, _))| para_points / para_epochs)
.filter(|(_, (_, para_epochs, _, _, _, _, _, _, _))| *para_epochs >= 1)
.map(|(_, (_, para_epochs, para_points, _, _, _, _, _, _))| {
para_points / para_epochs
})
.collect();
let max = avg_para_points.iter().max().unwrap_or_else(|| &0);
let min = avg_para_points.iter().min().unwrap_or_else(|| &0);
Expand All @@ -741,24 +754,30 @@ pub async fn get_validators(
//
aggregator_vec
.iter_mut()
.filter(|(_, (_, para_epochs, _, _, _, _, _))| *para_epochs >= 1)
.for_each(|(stash, (_, para_epochs, para_points, ev, iv, mv, s))| {
let mvr = *mv as f64 / (*ev + *iv + *mv) as f64;
let avg_para_points = *para_points / *para_epochs;
let score = if max - min > 0 {
(((1.0_f64 - mvr) * 0.75_f64
+ ((avg_para_points - *min) as f64 / (*max - *min) as f64) * 0.18_f64
+ (*para_epochs as f64 / total_epochs as f64) * 0.07_f64)
* 1000000.0_f64) as u32
} else {
0
};
*s = score;
// Add ranking stats to the validator result
validators.entry(stash.clone()).and_modify(|v| {
v.ranking = RankingStats::with(score, mvr, avg_para_points, *para_epochs);
});
});
.filter(|(_, (_, para_epochs, _, _, _, _, _, _, _))| *para_epochs >= 1)
.for_each(
|(stash, (_, para_epochs, para_points, ev, iv, mv, ba, bu, s))| {
let mvr = *mv as f64 / (*ev + *iv + *mv) as f64;
let bar = *ba as f64 / (*ba + *bu) as f64;
let avg_para_points = *para_points / *para_epochs;
let score = if max - min > 0 {
(((1.0_f64 - mvr) * 0.50_f64
+ bar * 0.25_f64
+ ((avg_para_points - *min) as f64 / (*max - *min) as f64)
* 0.18_f64
+ (*para_epochs as f64 / total_epochs as f64) * 0.07_f64)
* 1000000.0_f64) as u32
} else {
0
};
*s = score;
// Add ranking stats to the validator result
validators.entry(stash.clone()).and_modify(|v| {
v.ranking =
RankingStats::with(score, mvr, avg_para_points, *para_epochs);
});
},
);

// Filter by subset and min para epochs
// min_para_epochs = 1 if total_full_epochs < 12;
Expand All @@ -770,7 +789,7 @@ pub async fn get_validators(

let mut i = 0;
while i < aggregator_vec.len() {
let (_, (subset, para_epochs, _, _, _, _, _)) = &mut aggregator_vec[i];
let (_, (subset, para_epochs, _, _, _, _, _, _, _)) = &mut aggregator_vec[i];
if (*subset != params.subset && params.subset != Subset::NotDefined)
|| *para_epochs < min_para_epochs
{
Expand All @@ -781,8 +800,9 @@ pub async fn get_validators(
}

// Sort ranking validators by score
aggregator_vec
.sort_by(|(_, (_, _, _, _, _, _, a)), (_, (_, _, _, _, _, _, b))| b.cmp(&a));
aggregator_vec.sort_by(
|(_, (_, _, _, _, _, _, _, _, a)), (_, (_, _, _, _, _, _, _, _, b))| b.cmp(&a),
);

// Truncate aggregator
if params.size > 0 {
Expand Down Expand Up @@ -1183,10 +1203,28 @@ async fn calculate_validator_grade_by_stash(

let mvr = mvrs.iter().sum::<f64>() / para_epochs as f64;

// calculate bur if para_epochs > 0
let burs: Vec<f64> = data
.iter()
.filter(|v| v.is_para)
.filter_map(|v| v.para.get("bitfields"))
.filter_map(|value| <BitfieldsRecord as Deserialize>::deserialize(value).ok())
.map(|bitfields| {
let partial = bitfields.availability() + bitfields.unavailability();
if partial > 0 {
bitfields.unavailability() as f64 / partial as f64
} else {
0.0_f64
}
})
.collect();

let bur = burs.iter().sum::<f64>() / para_epochs as f64;

if params.show_summary {
return Ok(ValidatorGradeResult {
address: stash.to_string(),
grade: grade(1.0 - mvr).to_string(),
grade: grade(mvr, bur).to_string(),
authority_inclusion: auth_epochs as f64 / params.number_last_sessions as f64,
para_authority_inclusion: para_epochs as f64 / params.number_last_sessions as f64,
explicit_votes_total: data
Expand All @@ -1204,14 +1242,28 @@ async fn calculate_validator_grade_by_stash(
.filter(|v| v.is_para)
.map(|v| v.para_summary.missed_votes)
.sum(),
bitfields_availability_total: data
.iter()
.filter(|v| v.is_para)
.filter_map(|v| v.para.get("bitfields"))
.filter_map(|value| serde_json::from_value::<BitfieldsRecord>(value.clone()).ok())
.map(|bitfields| bitfields.availability())
.sum(),
bitfields_unavailability_total: data
.iter()
.filter(|v| v.is_para)
.filter_map(|v| v.para.get("bitfields"))
.filter_map(|value| serde_json::from_value::<BitfieldsRecord>(value.clone()).ok())
.map(|bitfields| bitfields.unavailability())
.sum(),
sessions_data: data.into(),
..Default::default()
});
}

return Ok(ValidatorGradeResult {
address: stash.to_string(),
grade: grade(1.0 - mvr).to_string(),
grade: grade(mvr, bur).to_string(),
authority_inclusion: auth_epochs as f64 / params.number_last_sessions as f64,
para_authority_inclusion: para_epochs as f64 / params.number_last_sessions as f64,
explicit_votes_total: data
Expand All @@ -1229,6 +1281,20 @@ async fn calculate_validator_grade_by_stash(
.filter(|v| v.is_para)
.map(|v| v.para_summary.missed_votes)
.sum(),
bitfields_availability_total: data
.iter()
.filter(|v| v.is_para)
.filter_map(|v| v.para.get("bitfields"))
.filter_map(|value| serde_json::from_value::<BitfieldsRecord>(value.clone()).ok())
.map(|bitfields| bitfields.availability())
.sum(),
bitfields_unavailability_total: data
.iter()
.filter(|v| v.is_para)
.filter_map(|v| v.para.get("bitfields"))
.filter_map(|value| serde_json::from_value::<BitfieldsRecord>(value.clone()).ok())
.map(|bitfields| bitfields.unavailability())
.sum(),
sessions: data.iter().map(|v| v.session).collect(),
..Default::default()
});
Expand Down
2 changes: 2 additions & 0 deletions src/api/responses.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -799,6 +799,8 @@ pub struct ValidatorGradeResult {
pub explicit_votes_total: u32,
pub implicit_votes_total: u32,
pub missed_votes_total: u32,
pub bitfields_availability_total: u32,
pub bitfields_unavailability_total: u32,
#[serde(skip_serializing_if = "Vec::is_empty")]
pub sessions: Vec<u32>,
#[serde(skip_serializing_if = "Vec::is_empty")]
Expand Down
Loading

0 comments on commit 9a702e9

Please sign in to comment.