The categorizer tier

The categorizer tier represents the machine-learning backend which we’re using to recognize images, and which will store the various models used for this purpose. Its job is to receive requests from the function tier, call a few MxNet routines, and return the response back to the function tier.

Let’s take a look at the class declaration:

class CategorizerTier : public mutils::ByteRepresentable,
                        public derecho::GroupReference {
protected:
    std::map<uint32_t, Model> raw_models;
    std::map<uint32_t, std::unique_ptr<InferenceEngine>> inference_engines;
    std::shared_mutex inference_engines_mutex;

public:
    CategorizerTier() {}
    CategorizerTier(std::map<uint32_t, Model>& _raw_models)
            : raw_models(_raw_models) {}
    ~CategorizerTier();

    Guess inference(const Photo& photo);

    int install_model(const uint32_t& tag, const ssize_t& synset_size,
                      const ssize_t& symbol_size, const ssize_t& params_size,
                      const BlobWrapper& model_data);
		      
    int remove_model(const uint32_t& tag);
    
    int ordered_install_model(const uint32_t& tag, const ssize_t& synset_size,
                              const ssize_t& symbol_size,
                              const ssize_t& params_size,
                              const BlobWrapper& model_data);

    int ordered_remove_model(const uint32_t& tag);

    REGISTER_RPC_FUNCTIONS(CategorizerTier, inference, install_model,
                           remove_model, ordered_install_model,
                           ordered_remove_model);

    DEFAULT_SERIALIZATION_SUPPORT(CategorizerTier, raw_models);
};

Much of this is reminiscent of the function tier; we’ll focus our investigation on those parts which differ.

The P2P entry points

The first thing to notice are the functions that we’ll call via the function tier:

    Guess inference(const Photo& photo);
    int install_model(const uint32_t& tag, const ssize_t& synset_size,
                      const ssize_t& symbol_size, const ssize_t& params_size,
                      const BlobWrapper& model_data);
    int remove_model(const uint32_t& tag);

Unlike in the gRPC code, these function entry points don’t take esoteric automatically-generated datatypes; instead, they simply take (by const reference) C++ types directly. The definitions of these functions are normal, exactly as they would have been defined for a normal local C++ class. This is an advantage of Derecho’s RPC framework over more general frameworks; by explicitly targeting only C++ code, we can take full advantage of C++ type information.

The ordered entry points

There are two more functions invoked via RPC in the categorizer tier:

    int ordered_install_model(const uint32_t& tag, const ssize_t& synset_size,
                              const ssize_t& symbol_size,
                              const ssize_t& params_size,
                              const BlobWrapper& model_data);
    int ordered_remove_model(const uint32_t& tag);

These functions serve as entry points for ordered_send, whereas the previous functions were designed to be called via p2p_send. In practice, the non-ordered versions of these methods simply perform an ordered_send in their body. This division into a p2p portion and an ordered portion is not fundamental to Derecho, but instead reflects the design of this demo.

Registering with Derecho

The last bit of code in the header, as before, corresponds to registering functions for RPC and specifying which fields need to be serialized:

    REGISTER_RPC_FUNCTIONS(CategorizerTier, inference, install_model,
                           remove_model, ordered_install_model,
                           ordered_remove_model);

    DEFAULT_SERIALIZATION_SUPPORT(CategorizerTier, raw_models);

Here we’ve decided to make inference, install_model, remove_model, ordered_install_model, and ordered_remove_model available for RPC, and we’ve chosen to serialize only the raw_models when this object is sent over the network.

Last updated on 22 Oct 2019
Published on 22 Oct 2019
Edit on GitHub