Home / Curated / Algorithms / FedOpt Adaptive Server Optimization (FedAdagrad, FedYogi, and FedAdam)

FedOpt Adaptive Server Optimization (FedAdagrad, FedYogi, and FedAdam)

Algorithms AIMech:FedOptAdaptiveServerOptimizationFedAdagradFedYogiFedAdam DRAFT

Aliases: FedOpt, FedAdagrad, FedYogi, FedAdam, Adaptive Federated Optimization

A general client-server federated optimization framework (FedOpt) and its adaptive server-side instantiations (FedAdagrad, FedYogi, and FedAdam) in which sampled clients i in S_t execute K steps of a stateless local optimizer ClientOpt (such as SGD with client learning rate eta_l) to compute model deltas Delta_i^t = x_{i,K}^t - x_t, and the server treats the negative average update -Delta_t = -(1/|S_t|) sum_{i in S_t} Delta_i^t as a pseudo-gradient fed into an adaptive per-coordinate server optimizer ServerOpt with server learning rate eta and adaptivity parameter tau > 0: updating momentum m_t = beta_1 m_{t-1} + (1 - beta_1) Delta_t, second-moment preconditioner v_t = v_{t-1} + Delta_t^2 (FedAdagrad), v_t = v_{t-1} - (1 - beta_2) Delta_t^2 sign(v_{t-1} - Delta_t^2) (FedYogi), or v_t = beta_2 v_{t-1} + (1 - beta_2) Delta_t^2 (FedAdam), and global iterate x_{t+1} = x_t + eta * m_t / (sqrt(v_t) + tau).

Identity Notes & Concept Scope Distinct from standard Federated Averaging (AIMech:FederatedAveragingClientServerOptimization, which corresponds to the special case where ServerOpt is SGD with unit learning rate eta = 1 and no server-side momentum or second-moment preconditioning) and from Centralized Yogi/Adam/Adagrad (which maintain optimizer state at every stochastic step rather than decoupling stateless local ClientOpt trajectories from stateful server-level pseudo-gradient preconditioning).

Claims & Grounded Evidence

4 Claims

Mechanism #r393-client-server-decoupling-and-zero-client-state-overhead

By confining first- and second-moment accumulators (m_t, v_t in R^d) exclusively to the central server in Algorithm 2 while running stateless local SGD on sampled clients, FedAdagrad, FedYogi, and FedAdam incur zero extra client-to-server communication bandwidth and avoid stale client-side preconditioner states in cross-device settings where clients participate rarely.

supports Source: Adaptive Federated Optimization · Locator: Section 2 General Framework & Section 3 Adaptive Federated Optimization (Algorithm 2), pp. 3-5

Mechanism #r393-nonconvex-convergence-rate-and-local-step-condition

Under L-smoothness, G-bounded gradients, local variance sigma_l^2, and inter-client gradient dissimilarity sigma_g^2 with effective variance sigma^2 = sigma_l^2 + 6 K sigma_g^2, Theorems 1-2 and Corollaries 1-2 prove that choosing client learning rate eta_l = Theta(1 / (K L sqrt(T))), server learning rate eta = Theta(sqrt(K m)), and tau = G / L achieves min_{0 <= t < T} E||nabla f(x_t)||^2 = O(1 / sqrt(m K T)) linear speedup in sampled clients m and local steps K whenever K = O(T sigma_l^2 / sigma_g^2).

supports Source: Adaptive Federated Optimization · Locator: Section 3.1 Convergence Analysis (Assumptions 1-3, Theorems 1-2, Corollaries 1-2), pp. 5-7

Limitation #r393-client-drift-inverse-local-learning-rate-limitation

Because local client updates drift under non-i.i.d. data (sigma_g^2 > 0), Theorem 3 shows that even on simple 1D quadratic client objectives, fixed client learning rates converge to a biased point whose offset scales with K * eta_l, requiring eta_l to scale inversely with K * sqrt(T) and joint tuning with the server adaptivity parameter tau.

supports Source: Adaptive Federated Optimization · Locator: Section 3.1 Theorem 2, Corollary 2, and Appendix B.1 Theorem 3, pp. 6-7, 15

Application #r393-cross-device-benchmark-suite-application

Evaluated across five realistic cross-device federated benchmarks—CIFAR-10, CIFAR-100, EMNIST-62, Shakespeare next-character RNN, and StackOverflow tag-prediction logistic regression and next-word NWP RNN—demonstrating that FedYogi and FedAdam significantly outperform FedAvg on heavy-tailed text vocabulary tasks and show wider learning-rate stability plateaus.

supports Source: Adaptive Federated Optimization · Locator: Section 4 Experiments and Section 5 Results (Figures 1-3, Table 2), pp. 7-9

⚠️ Curated Limitations & Negative Results

Requires decaying or O(1 / (K L sqrt(T))) client learning rates to prevent asymptotic client drift under non-i.i.d. heterogeneity.
Introduces server learning rate eta and adaptivity parameter tau that must be selected alongside client learning rate eta_l.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	narrow_match	FedOpt (FedAdagrad, FedYogi, FedAdam) is a specific adaptive server-side optimization methodology within FederatedLearning.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r393-primary-pdf`	Adaptive Federated Optimization Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecny, Sanjiv Kumar, H. Brendan McMahan	primary paper full_text	ID: https://doi.org/10.48550/arXiv.2003.00295 sha256:e955bf32183d…
`r393-arxiv-abs`	Adaptive Federated Optimization (arXiv abstract) Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecny, Sanjiv Kumar, H. Brendan McMahan	primary paper abstract_only	ID: https://arxiv.org/abs/2003.00295 sha256:3907f0649b5f…
`r393-metadata-cr`	Adaptive Federated Optimization (Crossref metadata) Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecny, Sanjiv Kumar, H. Brendan McMahan	primary paper abstract_only	ID: https://doi.org/10.48550/arXiv.2003.00295 sha256:fee30dfcfca4…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T07:56:00Z

Updated: 2026-10-19T07:56:00Z

Source File: data/curated/FedOptAdaptiveServerOptimizationFedAdagradFedYogiFedAdam.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories