AWS announced that Fortuna, an open-source toolkit for ML model uncertainty quantification, has been made generally available. Any trained neural network can be used with the calibration methods offered by Fortuna, such as conformal prediction, to produce calibrated uncertainty estimates.
There are numerous documented methods for estimating or calibrating the uncertainty of predictions, however current tools and libraries for quantifying uncertainty have a limited range and do not provide a comprehensive collection of methods. This has a large overhead and makes it difficult to incorporate uncertainty into production systems. Fortuna bridges this gap by compiling well-known techniques making them accessible to users through a standardized and user-friendly interface.
