and tune the Flower project you built in lesson two. You will learn about many aspects of federated systems that you may often want to tune, and you'll implement a custom training schedule in this lesson. All right, let's dive in. Compared to traditional centralized training, federated learning introduces some additional concepts, some additional components in the training process that can be customized and tune. One important factor is how do you select the clients that participate in each round? In this diagram you see five clients. So instead of sending the global model to all five clients, you could send a model to just three clients in the first round. The three clients would participate as usual. Then in the next round, you could select three other clients in a setting where there are only a few clients. The answer is often just select all clients in each round. If you have just five clients, as in this diagram, you would probably select all of them in each round of Federated learning. But then in settings where you have large numbers of clients, you would typically not do this. In fact, it has been shown that selecting more and more clients has diminishing returns. What you would do instead is you would select just a subset of available clients in a mobile setting with millions of clients. You would often just select a few hundred clients in any single round, or at most a few thousand, depending on the task. There are different strategies for selecting those clients. A very common one is to just select them randomly. There are also approaches that are, strictly speaking, not federated learning, but that can be very useful. One example is cyclic training, where you would send the model to one client, have that client train it, send it back to the server, and then send it to the next client. And you would do that for one client after another. Another factor is client configuration. Once you've selected clients, you need to decide how to configure them. What do you want the client to do? How long should you train the model? What hyperparameters should it use? Are there any other things the client needs to know to execute the training or evaluate in the right way? You can see the small model icons on the client highlighted. This is to indicate the client configuration that the client uses to execute the task. A third example for something that is commonly customized or tuned is aggregation. In federated learning, there are very different approaches for aggregating model parameters. You've seen and used federated averaging in the previous lesson. There are many other approaches like QFedAveraging or FedAdam that provide certain improvements over vanilla federated averaging. The Flower framework you used in the previous lesson has many of them built in as so-called strategies. Let's jump into the lab to see how the server can configure client side training. As usual, you start with some imports, in lesson one and two, You manually partitioned MNIST to simulate multiple data sets distributed across multiple user devices or multiple organizations. Instead of manually partitioning data among the participants, you can use a library called Flower Datasets. Flower datasets provides us with a class called Federated Dataset. This abstraction will partition many existing datasets like MNIST, and it allows you to generate small training and test sets for each client. The load data function loads and prepares the data for federated learning. It takes a partition IDs input, which specifies the partition of the data set to load the dataset used here is MNIST and it is partitioned into ten partitions. The load partition is then split into training and testing subsets with an 80 to 20 ratio. This is done using the train test split function. A custom transformation is applied to normalize the data. Data loaders are created for the training and testing subsets, respectively, using a PyTorch data loader. Along with model parameters, you often want to send configuration values to clients. Configuration values can be used for various purposes. Let's say you want a server to control the number of local epochs performed by each client, so the number of times the local client iterates over the local data set during training. To do that, you define a function called fit underscore config that takes only one argument. The current server round, and it returns a configuration dictionary. In the config dict, you put a key called local underscore epochs with integer value that tells the client how many local epochs to train for. In this example, you can see how you can vary that number depending on the current server round. Here, we send two in the first two rounds and then five for round three and later. The client can use this value to dynamically change the number of local epochs it trains on the local data set. It means that the client will train for two epochs initially, and then increase the number of local epochs during later rounds. Next, we initialize the federated averaging strategy as usual. We set fraction evaluate to zero, as we don't intend to perform any client-side evaluation, and we pass initial model parameters to a strategy. Now, to make the strategy use our new fit underscore config function, we simply pass it to fed averaging during initialization. Using the parameter on fit_config fn. Passing an unfit context function to federated averaging. We'll make the federated averaging strategy call this function every single round. The returned config dict will be included in the message that is sent to each client. It calls the function every time to enable you to send different configuration values to the client every single round. Last, we define an instance of server app You also create the flower client class as usual. The only difference is that you want to use the local underscore epochs value from the config dict. The fit method in Flower client class is responsible for training. It takes two main arguments, parameters, the model parameters from the server and config, to configuration parameters for training. As usual, the method begins by setting the parameters of the model using the provided parameters from the server. The number of local epochs, local underscore epochs, is extracted from the config dictionary to determine the number of epochs for local training. Here, you lock this value and then call the train underscore model function. The train underscore model function, is called to train the model using the client's local training data. This is stored in cells to train loader. This time we pass the number of local epochs as an additional argument. We then create the client function and client app as usual. Let's run server app and client app and see what happens. You can see that the Flower server app starts and it performs three rounds of federated learning. It initializes the global parameters that are provided by the strategy. And then it proceeds to the first round of federated learning. The strategy samples five clients out of five available clients, so it samples all of the available clients, and then it sends the config dictionary along with the model parameters, to all the participating clients. So to all of our five clients. You can see that, the first client logs that it trains for two epochs. So that's the second one. And so do all the other ones. After the training finishes, the aggregation function receives five results and zero failures. You can also see that, there's a log that says no clients selected skipping evaluation. This is because earlier on when initializing the federated averaging strategy we set friction, underscore evaluate to zero. So no evaluation is performed on the client side. In round two we see pretty much the same thing. We can see that, all of our clients, all of our five clients are selected. And you can see that each of those clients trains for two epochs. Now, looking at round three, you can see that again. Five clients out of five are selected. But then you can see that suddenly the clients, instead of training for just two epochs, they're training for five epochs. And this is caused by the configuration dictionary that those clients are receiving from the server. So the number of local epochs is controlled by the server, and the clients react to whatever configuration value they receive from the server, and they perform the appropriate number of local epochs. The configuration dictionary is a pretty flexible concept. You can put many different kinds of, keys and values in that dictionary, and it's a perfect thing to, experiment with. You could experiment with different hyperparameter schedules, for example. You can use it to set the learning rate, from the server to the client and control the learning rate that each client is supposed to use. And you can use it to control many different aspects of the client side in training process. Feel free to play with this and, experiment and let us know how it goes. Let's review lesson three. Federated learning introduces additional hyperparameters. It introduces additional concepts concepts that are important to control the training process. On the server side, we can customize and tune things like client selection, client configuration, and result aggregation. On the client side, we can configure things like the pre-processing, the local training, and any kind of post-processing we want to do on the weights before sending them back to the server. A Flower strategy allows you to customize all of those things that we talked about. On the client side, it allows you to customize client selection, client configuration, result aggregation, and server-side evaluation.