tower/retry/budget/
mod.rs

1//! A retry "budget" for allowing only a certain amount of retries over time.
2//!
3//! # Why budgets and not max retries?
4//!
5//! The most common way of configuring retries is to specify a maximum
6//! number of retry attempts to perform before giving up. This is a familiar idea to anyone
7//! who’s used a web browser: you try to load a webpage, and if it doesn’t load, you try again.
8//! If it still doesn’t load, you try a third time. Finally you give up.
9//!
10//! Unfortunately, there are at least two problems with configuring retries this way:
11//!
12//! **Choosing the maximum number of retry attempts is a guessing game.**
13//! You need to pick a number that’s high enough to make a difference when things are somewhat failing,
14//! but not so high that it generates extra load on the system when it’s really failing. In practice,
15//! you usually pick a maximum retry attempts number out of a hat (e.g. 3) and hope for the best.
16//!
17//! **Systems configured this way are vulnerable to retry storms.**
18//! A retry storm begins when one service starts to experience a larger than normal failure rate.
19//! This causes its clients to retry those failed requests. The extra load from the retries causes the
20//! service to slow down further and fail more requests, triggering more retries. If each client is
21//! configured to retry up to 3 times, this can quadruple the number of requests being sent! To make
22//! matters even worse, if any of the clients’ clients are configured with retries, the number of retries
23//! compounds multiplicatively and can turn a small number of errors into a self-inflicted denial of service attack.
24//!
25//! It's generally dangerous to implement retries without some limiting factor. [`Budget`]s are that limit.
26//!
27//! # Examples
28//!
29//! ```rust
30//! use std::sync::Arc;
31//!
32//! use futures_util::future;
33//! use tower::retry::{budget::{Budget, TpsBudget}, Policy};
34//!
35//! type Req = String;
36//! type Res = String;
37//!
38//! #[derive(Clone, Debug)]
39//! struct RetryPolicy {
40//!     budget: Arc<TpsBudget>,
41//! }
42//!
43//! impl<E> Policy<Req, Res, E> for RetryPolicy {
44//!     type Future = future::Ready<()>;
45//!
46//!     fn retry(&mut self, req: &mut Req, result: &mut Result<Res, E>) -> Option<Self::Future> {
47//!         match result {
48//!             Ok(_) => {
49//!                 // Treat all `Response`s as success,
50//!                 // so deposit budget and don't retry...
51//!                 self.budget.deposit();
52//!                 None
53//!             }
54//!             Err(_) => {
55//!                 // Treat all errors as failures...
56//!                 // Withdraw the budget, don't retry if we overdrew.
57//!                 let withdrew = self.budget.withdraw();
58//!                 if !withdrew {
59//!                     return None;
60//!                 }
61//!
62//!                 // Try again!
63//!                 Some(future::ready(()))
64//!             }
65//!         }
66//!     }
67//!
68//!     fn clone_request(&mut self, req: &Req) -> Option<Req> {
69//!         Some(req.clone())
70//!     }
71//! }
72//! ```
73
74pub mod tps_budget;
75
76pub use tps_budget::TpsBudget;
77
78/// For more info about [`Budget`], please see the [module-level documentation].
79///
80/// [module-level documentation]: self
81pub trait Budget {
82    /// Store a "deposit" in the budget, which will be used to permit future
83    /// withdrawals.
84    fn deposit(&self);
85
86    /// Check whether there is enough "balance" in the budget to issue a new
87    /// retry.
88    ///
89    /// If there is not enough, false is returned.
90    fn withdraw(&self) -> bool;
91}
tower/retry/budget/mod.rs

tower/retry/budget/
mod.rs
