Starting the server logic

Let’s take a look at the file src/derecho-component/server_logic.cpp:

#include <derecho-component/categorizer_tier.hpp>
#include <derecho-component/function_tier.hpp>
#include <derecho-component/server_logic.hpp>
#include <derecho/core/derecho.hpp>
#include <iostream>

void do_server(int argc, char** argv) {
    derecho::Conf::initialize(argc, argv);
    derecho::SubgroupInfo si{derecho::DefaultSubgroupAllocator(
            {{std::type_index(typeid(sospdemo::FunctionTier)),
              derecho::one_subgroup_policy(derecho::flexible_even_shards("FUNCTION_TIER"))},
             {std::type_index(typeid(sospdemo::CategorizerTier)),
              derecho::one_subgroup_policy(derecho::flexible_even_shards("CATEGORIZER_TIER"))}})};

    auto function_tier_factory = [](persistent::PersistentRegistry*) {
        return std::make_unique<sospdemo::FunctionTier>();
    };
    auto categorizer_tier_factory = [](persistent::PersistentRegistry*) {
        return std::make_unique<sospdemo::CategorizerTier>();
    };
    derecho::Group<sospdemo::FunctionTier, sospdemo::CategorizerTier> group(
            si, function_tier_factory, categorizer_tier_factory);
    std::cout << "Finished constructing Derecho group." << std::endl;
    std::cout << "Press ENTER to stop." << std::endl;
    std::cin.get();
    group.barrier_sync();
    group.leave(true);
}

This file is entirely concerned with building the derecho::Group object, which represents this node’s membership in the Derecho group, and provides the interface via which it can communicate with the other group members.

Let’s step through the code one line at a time:

    derecho::Conf::initialize(argc, argv);

Derecho accepts a large number of configuration parameters; almost every constant imaginable is tuneable to the users’ preferences. The derecho::Conf singleton class will read these parameters out of a configuration file located in the current working directory, making them available within the Derecho group construction logic.

    derecho::SubgroupInfo si{derecho::DefaultSubgroupAllocator(
            {{std::type_index(typeid(sospdemo::FunctionTier)),
              derecho::one_subgroup_policy(derecho::flexible_even_shards("FUNCTION_TIER"))},
             {std::type_index(typeid(sospdemo::CategorizerTier)),
              derecho::one_subgroup_policy(derecho::flexible_even_shards("CATEGORIZER_TIER"))}})};

The SubgroupInfo object contains all the information needed to assign nodes to various roles (or “subgroups”) within the overall Derecho system (or “top level group”). Much of this can be read from the Derecho configuration file; what we are specifying here is a SubgroupAllocator, which describes how to decide which nodes should belong to which subgroups. Derecho’s logic for doing this is incredibly flexible; users can effect arbitrary assignments, and can vary them as desired over time or based on environmental conditions.

In general, a Derecho top-level group is made up of a number of subgroups, each of which can be sharded for scalability. Subgroups are meant to represent single units of functionality (like “categorizer” or “router”); if you were to model a distributed system in an object-oriented language, you’d represent each subgroup with a single object. Leaning on this intuition, in Derecho all subgroups are backed by an instance of a user-provided Class. You can have any number of subgroups of each type, and every subgroup can be further divided into any number of shards.

In this particular example, we’ve chosen to set up a relatively simple group structure. We’ve decided that each subgroup type should have its own node-assignment logic, that there will be exactly one subgroup per named type, and that we should make sure all shards within that subgroup are evenly balanced among available nodes.

The first part of this is handled via the DefaultSubgroupAllocator. This takes a map from subgroup type to our desired allocation policy for that subgroup type. We’re choosing the one_subgroup_policy, and asking that nodes in that subgroup be assigned via flexible_even_shards. The final argument is the name of the relevant section of Derecho’s configuration file to consult for parameters for this subgroup or shard.

Constructing the group

The next pair of lines define “factories”: functions which are called to initialize each subgroup object.

    auto function_tier_factory = [](persistent::PersistentRegistry*) {
        return std::make_unique<sospdemo::FunctionTier>();
    };
    auto categorizer_tier_factory = [](persistent::PersistentRegistry*) {
        return std::make_unique<sospdemo::CategorizerTier>();
    };

For this example our factories aren’t very interesting; they just turn around and call constructors! But the design choice to use factories allows a level of flexibility which can’t be replaced by constructors alone. In particular, it allows us to capture any constructor arguments we will need for future invocations of this factory.

Finally we come to the construction of the group itself:

    derecho::Group<sospdemo::FunctionTier, sospdemo::CategorizerTier> group(
            si, function_tier_factory, categorizer_tier_factory);

All we’re doing here is passing the helper objects we’ve previously constructed into the group constructor. At this point, the constructor will initiate joining of the Derecho top-level group: it will spawn a new thread which will begin attempting to connect to the initial Derecho leader specified in the configuration file. This will not block the main thread for long; Derecho group construction and initialization will continue in the background as the main thread proceeds onto the next instruction.

    std::cout << "Finished constructing Derecho group." << std::endl;
    std::cout << "Press ENTER to stop." << std::endl;
    std::cin.get();
    group.barrier_sync();
    group.leave(true);

Finally we come to the end of the server initialization logic. This code simply prints out some informational messages and waits for the user to press enter. After the user has pressed enter, the code blocks at a top-level barrier until all other nodes in the Derecho group have reached the same barrier_sync call. This ensures we don’t try to leave the Derecho group while it is still under construction. Finally, we call leave, cleanly leaving the group.

If we had not called barrier_sync or had not explicitly called leave, the remainder of the Derecho top-level group would treat this node as though it had failed, and would recover so long as the remaining nodes represent a majority of the group. If the remaining nodes did not represent a majority, a partitioning exception would be thrown in the remaining alive nodes, intentionally crashing them. Note that a partitioning exception is not called in the event that all nodes leave after a barrier; Derecho understands that this is an intended shutdown event.

Group initialized

After the top-level Derecho group has been constructed, Derecho (following the information provided by users via the SubgroupInfo object) automatically assigns nodes to subgroups and constructs the subgroup objects. The system is now ready to receive requests via Derecho’s custom RPC operations, ordered_send and p2p_send.