Author

Tao Lin

738 papers · 1753–2026

1967peak 118 in 20262026

Frequent co-authors

Papers

showing 1–25 of 738 RSS

Ensemble Distillation for Robust Model Fusion in Federated Learning

Tao Lin, Lingjing Kong, Sebastian U. Stich +1 more

cs.LG · June 2020 · cited 1,610× · arXiv ↗ · more like this ·

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows…

Don't Use Large Mini-Batches, Use Local SGD

Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel +1 more

cs.LG · August 2018 · cited 477× · arXiv ↗ · more like this ·

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a \emph{post-local} SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of \emph{local SGD} variants.

Fog Orchestration for Internet of Things Services

Zhenyu Wen, Renyu Yang, Peter Garraghan +3 more

Computer Networks and Communications · IEEE Internet Computing · March 2017 · cited 333× · DOI ↗ · more like this ·

Large-scale Internet of Things (IoT) services such as healthcare, smart cities, and marine monitoring are pervasive in cyber-physical environments strongly supported by Internet technologies and fog computing. Complex IoT services are increasingly composed of sensors, devices, and compute resources within fog computing infrastructures. The orchestration of such applications can be leveraged to alleviate the difficulties of maintenance and enhance data security and system reliability. However, efficiently dealing with dynamic variations and transient operational behavior is a crucial challenge within the context of choreographing complex services. Furthermore, with the rapid increase of the scale of IoT deployments, the heterogeneity, dynamicity, and uncertainty within fog environments and increased computational complexity further aggravate this challenge. This article gives an overview…

Decentralized Deep Learning with Arbitrary Communication Compression

Anastasia Koloskova, Tao Lin, Sebastian U. Stich +1 more

cs.LG · July 2019 · cited 276× · arXiv ↗ · more like this ·

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD $-$ recently introduced and analyzed for strongly-convex objectives only $-$ converges under arbitrary high compression ratio on general non-convex functions at the rate $O\bigl(1/\sqrt{nT}\bigr)$ where $T$ denotes the number of iterations and $n$ the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over…

Dynamic Model Pruning with Feedback

Tao Lin, Sebastian U. Stich, Luis Barba +2 more

cs.LG · 12 Jun 2020 · cited 233× · arXiv ↗ · more like this ·

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.

A Joint Service Migration and Mobility Optimization Approach for Vehicular Edge Computing

Quan Yuan, Jinglin Li, Haibo Zhou +3 more

Computer Networks and Communications · IEEE Transactions on Vehicular Technology · 3 Jun 2020 · cited 177× · DOI ↗ · more like this ·

The vehicular edge computing is considered an enabling technology for intelligent and connected vehicles since the optimization of communication and computing on edge has a significant impact on driving safety and efficiency. In this paper, with the road traffic assignment to “proactively” reshape the spatiotemporal distribution of resource demands, we investigate the joint service migration and mobility optimization problem for vehicular edge computing. The goal is to meet the service delay requirements of vehicular edge computing with minimum migration cost and travel time. As service migration and mobility optimization are coupled, the joint scheduling problem suffers from the curse of dimensionality, which cannot be solved in real time by centralized algorithms. To this end, a multi-agent deep reinforcement learning (MADRL) algorithm is proposed to maximize the composite utility of…

Greenhouse Gas Emissions Accounting of Urban Residential Consumption: A Household Survey Based Approach

Tao Lin, Yunjun Yu, Xuemei Bai +2 more

Transportation · PLoS ONE · 6 Feb 2013 · cited 166× · DOI ↗ · more like this ·

Devising policies for a low carbon city requires a careful understanding of the characteristics of urban residential lifestyle and consumption. The production-based accounting approach based on top-down statistical data has a limited ability to reflect the total greenhouse gas (GHG) emissions from residential consumption. In this paper, we present a survey-based GHG emissions accounting methodology for urban residential consumption, and apply it in Xiamen City, a rapidly urbanizing coastal city in southeast China. Based on this, the main influencing factors determining residential GHG emissions at the household and community scale are identified, and the typical profiles of low, medium and high GHG emission households and communities are identified. Up to 70% of household GHG emissions are from regional and national activities that support household consumption including the supply of…

Measuring AI Ability to Complete Long Software Tasks

Thomas Kwa, Ben West, Joel Becker +23 more

cs.AI · March 2025 · cited 158× · arXiv ↗ · more like this ·

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon. This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024. The increase in AI models' time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and…

Intrusion Detection System Based on Decision Tree over Big Data in Fog Environment

Kai Peng, Victor C. M. Leung, Lixin Zheng +3 more

Computer Networks and Communications · Wireless Communications and Mobile Computing · 2018 · cited 157× · DOI ↗ · more like this ·

Fog computing, as the supplement of cloud computing, can provide low‐latency services between mobile users and the cloud. However, fog devices may encounter security challenges as a result of the fog nodes being close to the end users and having limited computing ability. Traditional network attacks may destroy the system of fog nodes. Intrusion detection system (IDS) is a proactive security protection technology and can be used in the fog environment. Although IDS in tradition network has been well investigated, unfortunately directly using them in the fog environment may be inappropriate. Fog nodes produce massive amounts of data at all times, and, thus, enabling an IDS system over big data in the fog environment is of paramount importance. In this study, we propose an IDS system based on decision tree. Firstly, we propose a preprocessing algorithm to digitize the strings in the given…

Revisiting Weighted Aggregation in Federated Learning with Neural Networks

Zexi Li, Tao Lin, Xinyi Shang +1 more

cs.LG · February 2023 · cited 147× · arXiv ↗ · more like this ·

In federated learning (FL), weighted aggregation of local models is conducted to generate a global model, and the aggregation weights are normalized (the sum of weights is 1) and proportional to the local data sizes. In this paper, we revisit the weighted aggregation process and gain new insights into the training dynamics of FL. First, we find that the sum of weights can be smaller than 1, causing global weight shrinking effect (analogous to weight decay) and improving generalization. We explore how the optimal shrinking factor is affected by clients' data heterogeneity and local epochs. Second, we dive into the relative aggregation weights among clients to depict the clients' importance. We develop client coherence to study the learning dynamics and find a critical point that exists. Before entering the critical point, more coherent clients play more essential roles in generalization.…

An Improved Analysis of Gradient Tracking for Decentralized Machine Learning

Anastasia Koloskova, Tao Lin, Sebastian U. Stich

cs.DC · 8 Feb 2022 · cited 146× · arXiv ↗ · more like this ·

We consider decentralized machine learning over a network where the training data is distributed across $n$ agents, each of which can compute stochastic model updates on their local data. The agent's common goal is to find a model that minimizes the average of all local loss functions. While gradient tracking (GT) algorithms can overcome a key challenge, namely accounting for differences between workers' local data distributions, the known convergence rates for GT algorithms are not optimal with respect to their dependence on the mixing parameter $p$ (related to the spectral gap of the connectivity matrix). We provide a tighter analysis of the GT method in the stochastic strongly convex, convex and non-convex settings. We improve the dependency on $p$ from $\mathcal{O}(p^{-2})$ to $\mathcal{O}(p^{-1}c^{-1})$ in the noiseless case and from $\mathcal{O}(p^{-3/2})$ to…

XCM: An Explainable Convolutional Neural Network for Multivariate Time Series Classification

Kevin Fauvel, Tao Lin, Véronique Masson +2 more

cs.LG · September 2020 · cited 145× · arXiv ↗ · more like this ·

Multivariate Time Series (MTS) classification has gained importance over the past decade with the increase in the number of temporal datasets in multiple domains. The current state-of-the-art MTS classifier is a heavyweight deep learning approach, which outperforms the second-best MTS classifier only on large datasets. Moreover, this deep learning approach cannot provide faithful explanations as it relies on post hoc model-agnostic explainability methods, which could prevent its use in numerous applications. In this paper, we present XCM, an eXplainable Convolutional neural network for MTS classification. XCM is a new compact convolutional neural network which extracts information relative to the observed variables and time directly from the input data. Thus, XCM architecture enables a good generalization ability on both large and small datasets, while allowing the full exploitation of…

On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm

Peng Sun, Bei Shi, Daiwei Yu +1 more

cs.CV · December 2023 · cited 144× · arXiv ↗ · more like this ·

Contemporary machine learning requires training large neural networks on massive datasets and thus faces the challenges of high computational demands. Dataset distillation, as a recent emerging strategy, aims to compress real-world datasets for efficient training. However, this line of research currently struggle with large-scale and high-resolution datasets, hindering its practicality and feasibility. To this end, we re-examine the existing dataset distillation methods and identify three properties required for large-scale real-world applications, namely, realism, diversity, and efficiency. As a remedy, we propose RDED, a novel computationally-efficient yet effective data distillation paradigm, to enable both diversity and realism of the distilled data. Extensive empirical results over various neural architectures and datasets demonstrate the advancement of RDED: we can distill the…

Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

Mengjie Zhao, Tao Lin, Fei Mi +2 more

cs.CL · April 2020 · cited 139× · arXiv ↗ · more like this ·

We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT and RoBERTa on a series of NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred simultaneously. Through intrinsic evaluations, we show that representations computed by masked language models encode information necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy. This confirms that masking can be utilized as an efficient alternative to finetuning.

Hybrid Neural Networks for Learning the Trend in Time Series

Tao Lin, Tian Guo, Karl Aberer

Signal Processing · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · 28 Jul 2017 · cited 139× · DOI ↗ · more like this ·

The trend of time series characterizes the intermediate upward and downward behaviour of time series. Learning and forecasting the trend in time series data play an important role in many real applications, ranging from resource allocation in data centers, load schedule in smart grid, and so on. Inspired by the recent successes of neural networks, in this paper we propose TreNet, a novel end-to-end hybrid neural network to learn local and global contextual features for predicting the trend of time series. TreNet leverages convolutional neural networks (CNNs) to extract salient features from local raw data of time series. Meanwhile, considering the long-range dependency existing in the sequence of historical trends of time series, TreNet uses a long-short term memory recurrent neural network (LSTM) to capture such dependency. Then, a feature fusion layer is to learn joint representation…

Training DNNs with Hybrid Block Floating Point

Mario Drumond, Tao Lin, Martin Jaggi +1 more

cs.LG · April 2018 · cited 131× · arXiv ↗ · more like this ·

The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP…

Evaluating Language-Model Agents on Realistic Autonomous Tasks

Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du +10 more

cs.CL · December 2023 · cited 124× · arXiv ↗ · more like this ·

In this report, we explore the ability of language model agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refer to this cluster of capabilities as "autonomous replication and adaptation" or ARA. We believe that systems capable of ARA could have wide-reaching and hard-to-anticipate consequences, and that measuring and forecasting ARA may be useful for informing measures around security, monitoring, and alignment. Additionally, once a system is capable of ARA, placing bounds on a system's capabilities may become significantly more difficult. We construct four simple example agents that combine language models with tools that allow them to take actions in the world. We then evaluate these agents on 12 tasks relevant to ARA. We find that these language model agents can only complete the easiest tasks from this list…

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Tao Lin, Sai Praneeth Karimireddy, Sebastian U. Stich +1 more

cs.LG · February 2021 · cited 123× · arXiv ↗ · more like this ·

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge and may severely deteriorate the generalization performance. In this paper, we investigate and identify the limitation of several decentralized optimization algorithms for different degrees of data heterogeneity. We propose a novel momentum-based method to mitigate this decentralized training difficulty. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10, ImageNet, and AG News) and several network topologies (Ring and Social Network) that our method is much more robust to the heterogeneity of clients' data than other existing methods, by a significant improvement in test performance ($1\% \!-\!…

Consensus Control for Decentralized Deep Learning

Lingjing Kong, Tao Lin, Anastasia Koloskova +2 more

cs.LG · February 2021 · cited 103× · arXiv ↗ · more like this ·

Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers from the degradation in the quality of the model: the training and test performance of models trained in a decentralized fashion is in general worse than that of models trained in a centralized fashion, and this performance drop is impacted by parameters such as network size, communication topology and data partitioning. We identify the changing consensus distance between devices as a key parameter to explain the gap between centralized and decentralized training. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. We…

On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

Chen Liu, Mathieu Salzmann, Tao Lin +2 more

cs.LG · June 2020 · cited 103× · arXiv ↗ · more like this ·

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.

No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier

Zexi Li, Xinyi Shang, Rui He +2 more

cs.LG · March 2023 · cited 94× · arXiv ↗ · more like this ·

Data heterogeneity is an inherent challenge that hinders the performance of federated learning (FL). Recent studies have identified the biased classifiers of local models as the key bottleneck. Previous attempts have used classifier calibration after FL training, but this approach falls short in improving the poor feature representations caused by training-time classifier biases. Resolving the classifier bias dilemma in FL requires a full understanding of the mechanisms behind the classifier. Recent advances in neural collapse have shown that the classifiers and feature prototypes under perfect training scenarios collapse into an optimal structure called simplex equiangular tight frame (ETF). Building on this neural collapse insight, we propose a solution to the FL's classifier bias problem by utilizing a synthetic and fixed ETF classifier during training. The optimal classifier…

Residential self-selection, built environment, and travel behavior in the Chinese context

Donggen Wang, Tao Lin

Transportation · Journal of Transport and Land Use · 11 Dec 2014 · cited 94× · DOI ↗ · more like this ·

Residential self-selection has been reported to be a factor confounding the observed relationship between built environment and travel behavior. By incorporating residential self-selection, studies have generated much insight into the causalities involved in the relationship between built environment and travel behavior. However, most of these studies were conducted in North American cities, where individuals may have the opportunity to realize their preferences in residential and transport mode choices. There are not many similar studies for other parts of the world, such as China, where residential and transport choices are probably more constrained than in North America. This paper aims to partly fill the gap by discussing the specificities of the residential self-selection issue in urban China and suggesting how to cope with this issue when examining the relationship between built…

Experimental demonstration of a skyrmion-enhanced strain-mediated physical reservoir computing system

Yiming Sun, Tao Lin, Na Lei +9 more

Artificial Intelligence · Nature Communications · 10 Jun 2023 · cited 78× · DOI ↗ · more like this ·

Abstract Physical reservoirs holding intrinsic nonlinearity, high dimensionality, and memory effects have attracted considerable interest regarding solving complex tasks efficiently. Particularly, spintronic and strain-mediated electronic physical reservoirs are appealing due to their high speed, multi-parameter fusion and low power consumption. Here, we experimentally realize a skyrmion-enhanced strain-mediated physical reservoir in a multiferroic heterostructure of Pt/Co/Gd multilayers on (001)-oriented 0.7PbMg 1/3 Nb 2/3 O 3 −0.3PbTiO 3 (PMN-PT). The enhancement is coming from the fusion of magnetic skyrmions and electro resistivity tuned by strain simultaneously. The functionality of the strain-mediated RC system is successfully achieved via a sequential waveform classification task with the recognition rate of 99.3% for the last waveform, and a Mackey-Glass time series prediction…

Mixed Chroma Sampling-Rate High Efficiency Video Coding for Full-Chroma Screen Content

Tao Lin, Peijun Zhang, Shuhui Wang +2 more

Signal Processing · IEEE Transactions on Circuits and Systems for Video Technology · 11 Oct 2012 · cited 77× · DOI ↗ · more like this ·

Computer screens contain discontinuous-tone content and continuous-tone content. Thus, the most effective way for screen content coding (SCC) is to use two essentially different coders: a dictionary-entropy coder and a traditional hybrid coder. Although screen content is originally in a full-chroma (e.g., YUV444) format, the current method of compression is to first subsample chroma of pictures and then compress pictures using a chroma-subsampled (e.g., YUV420) coder. Using two chroma-subsampled coders cannot achieve high-quality SCC, but using two full-chroma coders is overkill and inefficient for SCC. To solve the dilemma, this paper proposes a mixed chroma sampling-rate approach for SCC. An original full-chroma input macroblock (coding unit) or its prediction residual is chroma-subsampled. One full-chroma base coder and one chroma-subsampled base coder are used simultaneously to code…

Overview of Screen Content Video Coding: Technologies, Standards, and Beyond

Wen-Hsiao Peng, Frederick Walls, Robert Cohen +4 more

Signal Processing · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · December 2016 · cited 73× · DOI ↗ · more like this ·

This paper presents recent advances in screen content video coding, with an emphasis on two state-of-the-art standards: HEVC/H.265 Screen Content Coding Extensions (HEVC-SCC) by ISO/IEC Moving Picture Experts Group and ITU-T Video Coding Experts Group, and Display Stream Compression (DSC) by Video Electronics Standards Association. The HEVC-SCC enhances the capabilities of HEVC in coding screen content, while DSC provides lightweight compression for display links. Although targeting different application domains, they share some design principles and are expected to become the leading formats in the marketplace in the coming years. This paper provides a brief account of their background, key elements, performance, and complexity characteristics, according to their final specifications. As we survey these standards, we also summarize prior arts in the last decade and explore future…