Research / evaluation record
Graph-based feature engineering for DeFi security incident severity prediction
Published research followed by a stricter, reproducible evaluation of what the available graph evidence supports.
Reported conference result
0.598 → 0.787
AUC reported by the 2025 paper using the archival conference profile.
Corrected chronological holdout
0.823 / 0.817
Baseline / graph-enriched AUC under the timestamp-safe portfolio profile.
Research question
Does protocol context known before a DeFi security incident improve the ranking of incidents by severe financial loss?
The corrected experiment narrows the question to facts whose availability at prediction time can be demonstrated. Retrospective graph structure is not silently treated as historical evidence.
System view
One dataset. Two model variants.
01 Shared data
Incident dataset
One dataset, shared baseline inputs and the same preprocessing.
02 Model comparison
Baseline
Shared inputs
Graph-enriched
Shared inputs + timestamp-eligible graph features
03 Evaluation
Temporal validation
Chronological holdout and expanding-window cross-validation.
Identical splits for both modelsBoth models receive the same baseline inputs and deterministic LightGBM configuration. The enriched model adds only timestamp-eligible graph features.
Method
- Build the corrected dataset from 1,608 normalized incidents dated 2011–2025.
- Split observations chronologically at 2023-07-28 and fix the severe-loss threshold from the training period.
- Require every graph feature to have an availability timestamp no later than the incident.
- Evaluate a chronological holdout and five expanding-window temporal folds.
Results
Chronological holdout
AUC on the held-out period
Temporal cross-validation
Mean AUC ± standard deviation · 5 folds
| Model | Holdout AUC | F1 | Precision | Recall | Temporal CV AUC |
|---|---|---|---|---|---|
| Baseline | 0.823 | 0.545 | 0.611 | 0.491 | 0.775 ± 0.090 |
| Graph-enriched | 0.817 | 0.513 | 0.620 | 0.438 | 0.787 ± 0.074 |
Baseline
- Holdout AUC
- 0.823
- F1
- 0.545
- Precision
- 0.611
- Recall
- 0.491
- Temporal CV AUC
- 0.775 ± 0.090
Graph-enriched
- Holdout AUC
- 0.817
- F1
- 0.513
- Precision
- 0.620
- Recall
- 0.438
- Temporal CV AUC
- 0.787 ± 0.074
Graph enrichment did not improve the chronological holdout. Temporal CV was slightly higher on average, but variance and threshold sensitivity prevent a strong predictive claim.
Limits
- Three structural graph features are excluded because their historical availability is undocumented.
- Public incident coverage is incomplete and reported losses may be noisy or revised.
- Raw source records are not redistributed because redistribution rights have not been established.
- This is a research artifact, not a deployed risk-scoring or financial decision system.
Other research
01 record