Home / Curated / Algorithms / FedRep (Exploiting Shared Representations for Personalized Federated Learning)

FedRep (Exploiting Shared Representations for Personalized Federated Learning)

Algorithms AIMech:FedRepExploitingSharedRepresentationsForPersonalizedFederatedLearning DRAFT

Aliases: FedRep, Federated Representation Learning via Alternating Minimization-Descent, Exploiting Shared Representations for Personalized Federated Learning

A personalized federated learning algorithm (FedRep, Algorithm 1 and Algorithm 2) that decomposes each client i's model into a shared global low-dimensional representation phi: R^d -> R^k (with k << d) and a strictly local client-specific head h_i: R^k -> Y solving min_{phi in Phi} (1/n) sum_{i=1}^n min_{h_i in H} f_i(h_i o phi), alternating in each communication round between tau_h local gradient (or exact minimization) steps on the head h_i with phi^t frozen and tau_phi local gradient steps on the representation phi with the updated head h_i^{t,tau_h} frozen before server-side averaging of phi^{t+1}.

Identity Notes & Concept Scope Distinct from pFedMe (AIMech:PFedMePersonalizedFederatedLearningWithMoreauEnvelopes) and Ditto (AIMech:DittoFairAndRobustPersonalizedFederatedMultiTaskLearning), which regularize each client's full d-dimensional parameter vector toward a single global model via a Moreau-envelope or l2 penalty without explicitly decoupling a shared low-dimensional representation phi from client-specific heads h_i, from Per-FedAvg (AIMech:PerFedAvgModelAgnosticMetaLearningForPersonalizedFederatedLearning, which optimizes a MAML initial shared parameter vector across all layers), and from FedBN (AIMech:FedBnFederatedLearningOnNonIidFeaturesViaLocalBatchNormalization, which keeps only Batch Normalization layers local to address feature shift rather than heterogeneous label mappings).

Claims & Grounded Evidence

4 Claims

Mechanism #r404-exponential-subspace-convergence-and-collaborative-sample-complexity

Under linear representation models y_i^j = <w_i^*, B^{*T} x_i^j> + z_i^j with standard Gaussian covariates, client head diversity sigma_{min,*} = sigma_{min}((1/sqrt(n)) W^*) > 0, and principal angle distance dist(B, B^*) = ||B_perp^T B^*||_2 <= sqrt(1 - E_0^2), Theorem 1 proves that FedRep (Algorithm 2) contracts geometrically at rate dist(B^T, B^*) <= (1 - eta * E_0 * sigma_bar_{min,*}^2 / 4)^{T/2} dist(B^0, B^*) with per-client sample complexity m >= O(((kappa^4 k^2 d / (E_0^2 r n)) + (kappa^4 k^3 log(rn) / E_0^2)) log(1/epsilon)), reducing the d-dependent term by the total number of participating clients r*n.

supports Source: Exploiting Shared Representations for Personalized Federated Learning · Locator: Section 4 Theoretical Results (Definition 1, Assumptions 1-3, Theorem 1, Corollary 1), pp. 4-7

Mechanism #r404-fast-new-client-head-generalization-with-o-k-samples

Once an epsilon-accurate shared representation B^T with dist(B^T, B^*) <= epsilon is learned, a newly arriving client n+1 only needs to estimate its k-dimensional local head w_{n+1} in R^k (k << d), achieving excess prediction error O(epsilon^2 ||w_{n+1}^*||_2^2 + sigma^2 k / m_{n+1}) with only O(k) local samples instead of O(d).

supports Source: Exploiting Shared Representations for Personalized Federated Learning · Locator: Section 2.2 Comparison with Standard FL & Section 4.1 Discussion of Theorem 1, pp. 3-6

Limitation #r404-subgaussian-design-and-initialization-basin-limitation

Theoretical linear convergence guarantees in Theorem 1 are established for two-layer linear networks under isotropic Gaussian covariates x_i^j ~ N(0, I_d) and require an initial representation B^0 within principal angle distance sqrt(1 - E_0^2) of B^* (obtainable via federated spectral initialization).

supports Source: Exploiting Shared Representations for Personalized Federated Learning · Locator: Section 4.1 Assumptions 1-3, Theorem 1, and Remark 2 (Initialization), pp. 5-7

Application #r404-heterogeneous-cifar10-cifar100-femnist-and-sent140-application

Applied to personalized image classification and sentiment analysis across CIFAR-10, CIFAR-100, FEMNIST, and Sent140 with label-skewed client partitions, outperforming FedAvg, FedProx, Per-FedAvg, pFedMe, LG-FedAvg, and Ditto both on participating clients and on fine-tuned novel clients.

supports Source: Exploiting Shared Representations for Personalized Federated Learning · Locator: Section 5 Experiments (Table 1, Figures 1-4), pp. 7-9

⚠️ Curated Limitations & Negative Results

Theoretical subspace contraction guarantees are proven for linear representation models under Gaussian covariates.
Requires sufficient client diversity so that the true head matrix W^* spans the k-dimensional representation subspace.

Artificial Intelligence Ontology (AIO) Grounding

Mapped AIO Classes

AIO Target Class	Mapping Relation	Mapping Rationale
`AIO:FederatedLearning` Federated Learning	narrow_match	FedRep is a personalized FederatedLearning algorithm that decouples shared feature representations from local client-specific heads.

Evidence Sources & Primary Literature

3 Sources

Source ID	Publication / Document	Type / Access	Reference & Checksum
`r404-primary-pdf`	Exploiting Shared Representations for Personalized Federated Learning Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai	primary paper full_text	ID: https://proceedings.mlr.press/v139/collins21a.html sha256:9649bef7f3ff…
`r404-arxiv-abs`	Exploiting Shared Representations for Personalized Federated Learning (arXiv abstract) Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai	primary paper abstract_only	ID: https://arxiv.org/abs/2102.07078 sha256:9b9e1939c043…
`r404-metadata-cr`	Exploiting Shared Representations for Personalized Federated Learning (Crossref metadata) Liam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay Shakkottai	primary paper abstract_only	ID: https://proceedings.mlr.press/v139/collins21a.html sha256:a55e3ff8e424…

Ontology & Grounding Topology

Open in interactive network graph →

Curation Provenance

Curator: Antigravity

Method: ai_assisted

Model: Gemini

Created: 2026-10-19T08:07:00Z

Updated: 2026-10-19T08:07:00Z

Source File: data/curated/FedRepExploitingSharedRepresentationsForPersonalizedFederatedLearning.yaml · Schema: curated_concept.yaml

Explore across X-Mech

Related computational & biological Mechs: Embedding Map Network Graph ProteinTraitsMech (ESM-2 pLM) DUFMech (Domain pLM) TraitMech CultureMech All 14 X-Mech Repositories