In recent research, a team of researchers from Google Research presented FAX, an advanced software library built on top of JavaScript to improve computations used in federated learning (FL). It has been specifically developed to facilitate large-scale distributed and unified computing in various applications, including data center and cross-device situations.

Using the sharing features of JAX, FAX enables seamless integration with TPUs (Tensor Processing Units) and sophisticated JAX runtimes such as Pathways. It provides several important benefits by directly embedding the necessary building blocks for federated computations as primitives within JAX.

The library provides extensibility, simple JIT compilation and AD capabilities. In FL, clients work together on Machine Learning (ML) assignments without revealing their personal information, and unified computations often involve multiple client training models simultaneously while maintaining periodic synchronization. On-device clients can be used in FL applications, but high-performance data center software is still necessary.

FAX overcomes these issues by offering a framework for defining scalable distributed and federated computing in data centers. Through its Primitive mechanism, it incorporates a federated programming model into JAX, allowing FAX to make use of JIT compilation and parsing in XLA.

FAX has the ability to share computations between models and clients, as well as intra-client data between logical and physical mesh devices. It uses innovations in distributed data center training such as Pathways and GSPMD. The team shared that FAX can also provide Federated Automatic Differentiation (federated AD) by facilitating forward and reverse differentiation through JAX’s primitive mechanism. This allows data location information to be preserved during the differentiation process.

The team summarized their main contributions as follows.

1. XLA HLO (XLA High-Level Optimizer) format translation of FAX calculations is efficient. A domain-specific compiler called XLA HLO prepares computational graphics for use with a range of hardware accelerators. Through the use of this feature, FAX can fully utilize hardware accelerators such as TPU, leading to improved performance and performance.

2. A thorough implementation of federal automated differentiation has been included in the FAX. This feature automates the gradient computation process through the complex organization of federated learning, greatly simplifying the expression of federated computations. FAX speeds up the process of automatic differentiation, which is a critical part of training ML models, especially for federated learning tasks.

3. FAX calculations are made to work easily with federal inter-device computing systems currently in use. This means that computations created by facsimile, whether involving data center servers or on-device clients, can be quickly and simply deployed and performed in real-world federated learning environments.

In conclusion, FAX is flexible and can be used for various ML computations in data centers. Beyond FL, it can handle a wide range of distributed and parallel algorithms, such as FedAvg, FedOpt, branch-train-merge, DiLoCo, and PAPA.

check it Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us Twitter. Join us Telegram channel, Discord Channeland LinkedIn Groops.

If you like our work, you will love our work newsletter..

Don’t forget to join us 38k+ ML SubReddits