SUM — Surgery & Merge for Federated Class-Incremental Learning

Abstract

Real-world intelligent systems often require both distributed collaboration across data-isolated clients and continual adaptation to evolving tasks. This setting naturally gives rise to Federated Class-Incremental Learning (FCIL), which combines Federated Learning (FL) and Continual Learning (CL). However, their combination introduces two coupled sources of interference: spatial interference from heterogeneous clients and temporal interference from sequential tasks, jointly leading to Spatial–Temporal Catastrophic Forgetting (ST-CF). Existing approaches typically address spatial and temporal interference with separate mechanisms, often incurring additional client-side computation or communication, while leaving directional interactions among updates during aggregation unregulated. In this paper, we reinterpret FCIL as a unified multi-task learning problem, where both client and task updates are represented as adaptation vectors in a shared parameter space. Based on this view, we propose Surgery & Merge (Sum), a purely server-side framework that performs geometric surgery on adaptation vectors during aggregation. Spatial Sum mitigates client-level interference within each round, while causal online temporal Sum removes cross-task interference over time without additional client-side computation, communication, or memory beyond standard federated training. Empirically, Sum achieves up to 22% improvement over prior FCIL methods across diverse vision and language benchmarks while remaining robust to unreliable clients and maintaining computational efficiency.

18 / 18best vision settings

up to 22%gain over prior FCIL methods

server-onlyno extra client-side cost

01

Motivation

Directional interactions decide what survives aggregation.

FCIL couples two failure modes: non-IID clients create spatial interference inside each communication round, while sequential tasks create temporal interference over the learning history. Treating them independently leaves residual interference on the other axis.

In the shared parameter space, both client and task updates act as adaptation vectors. Aligned components can repeat the same update; opposing components can erase useful knowledge. SUM therefore regulates their geometry at the point where those vectors actually meet: server aggregation.

Adaptation vectors and examples of redundant and destructive directional interactions.

Aggregation is not a passive average—it is the control point for what the global model retains.

02

Method

One surgery principle, applied across space and time.

SUM keeps standard client training and parameter communication intact. The server first refines concurrent client updates within each round, then causally refines each completed task against the accumulated task history and stores a compact inference module.

Two-task lifecycleTask 1 · communication round 1 / 5

Task 1Spatial SUM every round

1R1

2R2

3R3

4R4

5R5

ΣSurgery

Task 2Spatial SUM every round

1R1

2R2

3R3

4R4

5R5

ΣSurgery

✓Inference
ready

Spatial SUM regulates client interactions within every communication round.

Trim: Z-score statistics suppress extreme coordinates before vector comparison.
Surgery: pairwise projection removes redundant and destructive client-coupled components.
Merge: the server aggregates only refined residuals; local training and uploads remain unchanged.

03

Experiments

SUM leads across every evaluated vision setting.

Final Averaged Accuracy (FAA, ↑) under the same backbone, task split, client partition, communication rounds, and data-access constraints. Each dataset is evaluated at three levels of client heterogeneity.

Main quantitative results

Best values are highlighted.

Method	CIFAR-100			ImageNet-R			ImageNet-A			EuroSAT			CARS-196			CUB-200
Method	.5	.1	.05	.5	.1	.05	1.0	.5	.2	1.0	.5	.2	1.0	.5	.2	1.0	.5	.2
Joint	92.75			84.02			54.64			98.42			85.62			86.04
EWC	78.46	72.42	64.51	58.93	48.15	43.68	10.86	10.07	8.89	64.12	59.30	56.52	19.55	18.02	18.29	31.46	29.60	27.89
LwF	62.87	55.56	47.09	54.03	41.02	46.07	8.89	8.89	7.90	31.91	21.26	31.42	20.84	22.72	31.76	25.25	21.11	18.54
FisherAvg	76.10	74.43	65.31	58.68	50.82	47.33	11.59	11.06	10.14	58.84	59.94	55.86	26.03	24.60	21.58	30.45	28.39	25.06
RegMean	59.80	45.88	39.08	61.18	57.00	55.80	8.56	6.22	4.34	48.74	51.73	45.27	21.83	20.36	15.92	35.57	32.84	32.83
CCVR	79.95	75.14	65.30	70.00	62.60	60.38	39.50	36.27	35.94	64.44	57.93	62.69	38.99	37.81	35.31	62.67	59.48	56.33
L2P	83.88	61.54	55.00	42.08	23.85	16.98	20.14	17.31	16.85	40.63	51.78	45.46	35.49	31.00	20.01	56.23	47.31	38.16
CODA-P	82.25	61.82	46.74	61.18	36.73	25.82	18.30	14.48	7.31	73.38	69.42	66.69	28.04	20.83	14.53	42.53	37.71	29.19
FedProto	75.79	70.02	60.55	58.52	47.30	52.93	9.87	9.22	10.01	58.79	62.85	64.17	26.08	24.55	22.75	30.22	28.27	26.01
TARGET	74.72	72.32	62.60	54.65	45.83	41.32	10.27	11.39	10.73	52.74	52.74	45.11	28.65	27.20	26.13	39.30	38.40	34.79
PILoRA	76.48	75.81	74.80	53.67	51.62	49.37	19.62	18.70	20.01	48.35	32.89	31.22	37.57	37.92	36.95	61.11	60.68	60.39
FOT	79.26	76.45	69.89	68.65	59.92	55.12	18.37	17.25	15.01	59.58	57.07	48.07	33.53	33.49	30.57	45.88	41.78	41.28
LoRM	86.95	81.75	82.76	72.48	63.83	66.45	37.26	36.34	33.11	84.23	77.26	81.36	54.41	51.87	48.81	64.60	63.67	60.06
SUM	97.03	96.93	96.32	83.80	85.58	86.81	49.90	50.82	47.86	96.51	93.28	98.57	68.78	68.83	67.57	81.69	79.79	79.43

Learning dynamics

Fast adaptation without progressive forgetting.

Across FCIL baselines, each task begins with a pronounced V-shaped drop before the model recovers over the five communication rounds. Their task-end accuracy also declines as more tasks arrive, revealing accumulated catastrophic forgetting.

SUM shows a much shallower within-task V-shape and maintains a nearly flat task-end envelope. Spatial surgery stabilizes adaptation inside each task, while temporal surgery preserves what earlier tasks contributed.

Learning progression plot comparing SUM with FCIL baselines.

The curve separates two effects: faster recovery within tasks and stronger retention across tasks.

04

Analysis

Every stage contributes—and SUM scales smoothly.

Removing spatial surgery produces the largest drop, but every component contributes to the final result. The ablation isolates trimming, spatial and temporal refinement, sparsification, sign election, and mask-based modularization under the same CIFAR-100 setting.

Component ablation

CIFAR-100, β = 0.05

Module	Ablation setting	FAA ↑	Δ	Wall time
Spatial SUM	w/o Z-score Trimming	85.87	−10.45	1.00×
Spatial SUM	w/o Spatial Surgery	78.93	−17.39	0.77×
Temporal SUM	w/o Temporal Surgery	87.67	−8.65	0.90×
Inference-ready	w/o Sparsification	90.23	−6.09	0.92×
	w/o Elect-Sign	94.58	−1.74	0.93×
	w/o Mask Modular	91.24	−5.08	1.00×
SUM	Full method	96.32	—	1.00×

Surgery target & scalability

Bidirectional surgery is consistently stronger than removing only positive or only negative interactions: redundant and destructive components must be handled together. The gain also survives deployment-scale changes. As the number of clients grows, SUM remains near or above Joint while EWC degrades and LoRM plateaus; as ViT size increases from Tiny to Large, SUM continues to scale and stays ahead of the centralized reference. These trends indicate that the improvement is not tied to one client count or one backbone capacity.

Surgery target, client scalability, and model scalability analyses.

BibTeX

@article{kim2026sum,
  title   = {SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class-Incremental Learning},
  author  = {Kim, Jaeik and Do, Jaeyoung},
  journal = {arXiv preprint arXiv:2607.19384},
  year    = {2026}
}

SUM: Unified Geometric Surgery on Spatio-Temporal
Adaptation Vectors for Federated Class-Incremental Learning

Abstract

Directional interactions decide what survives aggregation.

One surgery principle, applied across space and time.

Spatial SUM regulates client interactions within every communication round.

Temporal SUM protects accumulated task knowledge at each task boundary.

The inference-ready module stores task identity without a full model copy.

SUM leads across every evaluated vision setting.

Main quantitative results

Fast adaptation without progressive forgetting.

Every stage contributes—and SUM scales smoothly.

Component ablation

Surgery target & scalability

BibTeX

SUM: Unified Geometric Surgery on Spatio-TemporalAdaptation Vectors for Federated Class-Incremental Learning

Abstract

Spatial SUM regulates client interactions within every communication round.

Temporal SUM protects accumulated task knowledge at each task boundary.

The inference-ready module stores task identity without a full model copy.

Main quantitative results

Fast adaptation without progressive forgetting.

Component ablation

Surgery target & scalability

BibTeX

SUM: Unified Geometric Surgery on Spatio-Temporal
Adaptation Vectors for Federated Class-Incremental Learning