The architecture of libkazv

libkazv is a sans-io library based on lager implementing the Matrix client-server API.

lager, enabling the unidirectional data flow

You should probably read the lager docs first: https://github.com/arximboldi/lager.

Model, Action, Reducer

The basics of lager is that you can think of it as a state machine. The model is a value that represents the application state.

struct Model {
    /* ... */
};
 
Model current;

Actions are values that describe changes that are to be made to the model. Usually, there are more than one kind of actions, and we can unify them with a variant:

struct A1 { ... };
...
 
using Action = std::variant<A1, A2, ...>;
Action action = A1{...};

A reducer is a function taking a model and an action, and returns another model and an effect. The reducer applies the changes described in the action onto the model:

// The reducer function is by convention called update()
auto [next, effect] = update(current, action);

The effect is then executed against a context to perform side-effects:

effect(context);

Effects are optional, though. The reducer may also return only a model and no effect.

auto next = update(current, action);

Store, Context, Deps

There is a layer above this: the store contains the model and tracks the changes to it. The store constructor takes an event loop, which accepts posts of functions. They are run in a queue, and no more than one post can run at the same time. The time frame in which one post is run is called a tick of the event loop.

auto initialModel = ...;
auto eventLoop = ...;
auto store = makeStore(
    initialModel,
    withEventLoop(eventLoop),
    ...);

A store contains a context which effects are executed against. Contexts can be used to dispatch actions. A dispatch schedules (posts) the following in one tick of the event loop:

• Run the reducer
• Update the value of the model
• Execute the effect against the context.

Here is roughly what happens when an action is dispatched:

post([=]() {
    auto [next, effect] = update(current, action);
    setCurrent(next);
    effect(context);
});

The store can contain dependencies when it is created, and these dependencies are available in the context as well:

auto store = makeStore(..., lager::with_deps(depA, depB));

These dependencies can then be retrieved from the context in the effects. This is commonly used to perform IO:

std::pair<Model, Effect> update(Model m, SomeAction a)
{
    auto nextModel = ...;
    auto request = getRequest(a);
    auto effect = [request](const auto &ctx) {
        auto ioHandler = std::get<IOHandler>(ctx);
        ioHandler.httpRequest(request);
    };
    return std::make_pair(nextModel, effect);
}

Cursors and Lenses

On the other hand, the current value of the model can be obtained from the store via a cursor. For the purpose of the Kazv Project, the most-used cursor is a read-only one called lager::reader. The get() method on the cursor returns the current value stored in it.

lager::reader<Model> cursor = store;
Model model = cursor.get();

Cursors can be watched. When the value in a cursor changes, the watch callback is invoked (in the event loop):

lager::watch(cursor, [](auto newModel) {
    ...
});

Cursors can also be derived. There are plenty of derivations possible, but the most used ones are lenses and map-transformations. Map-transformations are simple, it takes a function and returns another cursor:

struct Model {
    int a;
    int b;
};
 
lager::reader<Model> cursor = ...;
lager::reader<int> cursorA = cursor.map([](Model m) {
    return m.a;
});

Lenses are relationships between two values (usually a whole and a part). For the purpose of libkazv, you can see it as a similar transformation as map, but more concise. Commonly used lenses include:

• Attribute lenses:

  struct Model {
      int a;
      int b;
  };
   
  lager::reader<Model> cursor = ...;
  lager::reader<int> cursorA = cursor[&Model::a];

• at lenses:

  lager::reader<std::vector<int>> cursor = ...;
  lager::reader<std::optional<int>> cursor0 = cursor[0];

lager::reader<std::unordered_map<std::string, int>> cursor = ...;
lager::reader<std::optional<int>> cursorA = cursor["A"s];

• Lenses about optional:

  lager::reader<std::optional<int>> cursor = ...;
  lager::reader<int> cursorI = cursor[lager::lenses::value_or(42)];
  lager::reader<int> cursorI = cursor[lager::lenses::or_default];

Cursors can be combined with lager::with():

lager::reader<int> a = ...;
lager::reader<int> b = ...;
lager::reader<int> c = lager::with(a, b).map(
    [](int aValue, int bValue) {
        return aValue + bValue;
    });

Thread-safety

Note that all cursor operations (get() and transformations) need to happen in the thread of the event loop. (see https://github.com/arximboldi/lager/issues/118) If you need to operate on cursors in another thread, you need to create a secondary root. This is done by creating another store with an event loop running in the desired thread, with the action type being the same as the model type, and then watching the primary root:

auto secondaryStore = makeStore(
    initialModel,
    lager::with_reducer([](Model, Model next) { return next; }),
    ... // pass an event loop running in the desired thread
);
lager::watch(mainStore, [ctx=secondaryStore.context()](Model m) {
    ctx.dispatch(std::move(m));
});
 
// in the thread of the secondary store's event loop
auto cursor = secondaryStore.map(...);

This pattern can be used when, for example, the primary store's event loop is running in a worker thread, and the data needs to be passed to the UI thread.

sans-io

libkazv is sans-io, as it has a business logic layer (the models/reducers in kazvclient and kazvcrypto) and an abstract IO layer (Sdk/Client/Room, JobInterface/EventInterface/PromiseInterface).

How lager is used in libkazv

The models (SdkModel, ClientModel, RoomModel, etc.) represent the state of the client. Kazv::ClientModel not only contains the application state, but also describes temporary things such as which jobs (network requests) to be sent and which triggers (e.g. notifications) to be emitted.

The out-most Kazv::SdkModel provides an interface to save and load the state with boost::serialization. Its reducer also takes out Kazv::ClientModel::nextJobs and Kazv::ClientModel::nextTriggers, and send them in the effect returned.

libkazv uses a stronger store (Kazv::Store) than lager's builtin store, allowing then-continuations (see https://github.com/arximboldi/lager/issues/96). We take the concept of Promises from Promises/A+. It is similar to C++'s std::future. A Promise can be then-ed with a continuation callback, which is executed after the Promise resolves. A Promise can contain data, and the Promise is said to have resolved when the data is ready. For the case of libkazv, the data contained is of type Kazv::EffectStatus.

auto promise = ctx.dispatch(action);
promise.then([](auto status) {
    // this will be run after the reducer for action is run,
    // and after the effect returned by the reducer is run.
    // `status` will contain the value returned from the effect.
});

The context can create a resolved Promise:

ctx.createResolvedPromise(EffectStatus(...));

A Promise can be created by the context (Kazv::Context) through a callback:

ctx.createPromise([](auto resolve) {
    // do some heavy computations or IO
    resolve(value);
}); // this promise will resolve after `resolve` is called

The resolve() function can be also called with another Promise. This creates a Promise that is set to wait for that Promise. A Promise, p1, that waits for another Promise p2 will resolve after p2 has resolved, and will contain the same data as p2.

ctx.createWaitingPromise([](auto resolve) {
    // ...
    auto anotherPromise = ...;
    resolve(anotherPromise);
});

Moreover, in libkazv's store, the effect can return a Promise as well. If this is the case, the Promise returned from the dispatch will be set to wait for the Promise returned from the effect.

std::pair<Model, Effect> update(Model m, Action a)
{
    return {nextModel, [](const auto &ctx) {
        auto ioHandler = getIOHandler(ctx);
        auto p = ctx.createPromise([](auto resolve) {
            ioHandler.request(..., [](auto value) {
                auto data = doSomethingWith(value);
                return resolve(data);
            });
        });
        return p;
    }};
}
 
auto promise = ctx.dispatch(Action{}); // will be resolved after p has resolved

Actually, this is what exactly the reducer of Kazv::SdkModel (Kazv::SdkModel::update) does: it returns an effect that returns a Promise that resolves after all the jobs in Kazv::ClientModel::nextJobs have been sent and we get a response for each of them.

How to add an action (API request) to libkazv

Action and Job

First, choose an action name, and add it to src/client/clientfwd.hpp . You need to add both the forward declaration and the Kazv::ClientAction variant. They are grouped by types so try to add it next to related actions.

Then, think of what parameters the action takes, and add the struct definition to src/client/client-model.hpp . If this is an API request, it should usually contain all the parameters by the corresponding job constructor in Kazv::Api (except for the server url and the access token, which are already given in the ClientModel.

Start writing a reducer for your action. In libkazv, the reducer for ClientActions lies in src/client/actions . They are also grouped by types into files, so choose a file with relevant actions to begin with. If there are none, create a new file. In the hpp file, write at least two function signatures:

namespace Kazv
{
    ClientResult updateClient(ClientModel m, XXXAction a);
    ClientResult processResponse(ClientModel m, YYYResponse r);
}

where XXXAction is the name of your action, and YYY is the name of the job in Kazv::Api. For example, the response type of LoginJob is LoginResponse. Be sure to include relevant headers in src/api/csapi .

Reducer, part 1 (or: How to send the request)

In the cpp file, write the definitions of the updateClient() function. It needs to add a job from the action.

namespace Kazv
{
    ClientResult updateClient(ClientModel m, XXXAction a)
    {
        // maybe verify the parameters and return an effect
        // that returns a failed EffectStatus
        // do this only if needed
        if (/* it has invalid parameters */) {
            return { m, failEffect("MOE.KAZV.MXC_SOME_ERROR_CODE", "some error message") };
        }
 
        auto job = m.job<YYYJob>().make(
            a.param1, a.param2, ...
        );
        m.addJob(std::move(job));
        return { m, lager::noop };
    }
}

Write a unit test to ensure the correct request is made. This is important because we auto-generate the C++ code for the jobs, and the order of the parameters in the job constructor can differ from version to version of the Matrix spec docs. We use Catch2-3 for unit testing.

// src/tests/client/xxx-test.cpp
#include "client/actions/xxx.hpp"
 
using namespace Kazv;
using namespace Kazv::Factory;
 
TEST_CASE("XXXAction", "[client][xxx]") // add test tags as needed
{
    auto client = makeClient(); // see design-docs/libkazvfixtures.md
    auto [next, _] = updateClient(client, XXXAction{...});
    assert1Job(next);
    for1stJob(next, [](const BaseJob &job) {
        REQUIRE(job.url().find(...) != std::string::npos);
        auto body = json::parse(std::get<BytesBody>(job.requestBody()));
        REQUIRE(body.at("param1") == ...);
        REQUIRE(body.at("param2") == ...);
    });
}

If needed, add the test cpp file to src/tests/CMakeLists.txt.

Reducer, part 2 (or: How to process the response)

The next step is to think about how to process the response returned from the server. Does it need to modify any client state, or will the state change be returned from sync? In either case, you need to do error handling. This can be done conveniently with helper functions.

// src/client/actions/xxx.cpp
 
namespace Kazv
{
    ClientResult processResponse(ClientModel m, YYYResponse r)
    {
        if (!r.success()) {
            return { std::move(m), failWithResponse(r) };
        }
 
        // here the response is successful
        // modify client state as needed
        ...;
        return { std::move(m), lager::noop };
        // Or, if some data from the response needs to be returned to the user, use:
        // return { std::move(m), [=](const auto &ctx) {
        //     return EffectStatus(/* succ = */ true, json{
        //         {"result1", result1},
        //     });
        // }};
    }
}

It is important to add the processing of the response to src/client/client-model.cpp , in the reducer for ProcessResponseAction.

[&](ProcessResponseAction a) -> Result {
    auto r = ...;
    // ...
 
    RESPONSE_FOR(YYY);
    // ...
}

If you find that processing the response is only possible if you have access to the data that is not present in the response, but only present in the action, you may add additional data to the job in the updateClient() function:

ClientResult updateClient(ClientModel m, XXXAction a)
{
    auto job = (...).withData(json{
        {"param1", a.param1},
    });
    m.addJob(std::move(job));
    // ...
}

And then the response will contain the data json from the corresponding job:

ClientResult processResponse(ClientModel m, YYYResponse r)
{
    // ...
    auto param1 = r.dataStr("param1"); // this gives data["param1"] converted to std::string
    auto param2 = r.dataJson("param2"); // this gives data["param2"] as json
    // ...
}

Similarly, add a test to verify it has the correct handling:

TEST_CASE("Process YYY response", "[client][xxx]")
{
    boost::asio::io_context io;
    AsioPromiseHandler ph{io.get_executor()};
    auto initialModel = makeClient(...);
 
    auto store = createTestClientStoreFrom(initialModel, ph);
    auto client = Client(store.reader().map([](auto c) { return SdkModel{c}; }), store, std::nullopt);
 
    WHEN("Success response")
    {
        auto succResponse = makeResponse("YYY");
        // or, if data is needed:
        // auto succResponse = makeResponse("YYY", withResponseDataKV("param1", value) | withResponseKV(...));
        store.dispatch(ProcessResponseAction{succResponse})
            .then([client] (auto stat) {
                REQUIRE(stat.success());
                // validate that the client is modified accordingly
                REQUIRE(client.someGetter().make().get() == ...);
            });
    }
 
    WHEN("Failed response")
    {
        auto failResponse = makeResponse("YYY", withResponseJsonBody(R"({
  "errcode": "ERROR_CODE",
  "error": "error message"
})"_json) | withResponseStatusCode(403)); // replace with the response body and error code specified in the matrix spec
        store.dispatch(ProcessResponseAction{failResponse})
            .then([] (auto stat) {
                REQUIRE(!stat.success());
                REQUIRE(stat.dataStr("error") == ...);
                REQUIRE(stat.dataStr("errorCode") == ...);
            });
    }
 
    io.run();
}

And from here you are done.

Here is a real-world example of adding an action to libkazv: https://lily-is.land/kazv/libkazv/-/merge_requests/78/diffs.