Understand gRPC

We control the Decidim instance through our Strapi backend. To establish a connection between the backend and Decidim, we incorporate functionality (a Ruby gem) into the Decidim codebase, exposing a private API within a private network. This technique enables us to expose an internal API for Decidim, accessible exclusively by our Strapi backend.

Internal API for decidim: gRPC

From voca, we need to execute remote procedures, which are sometimes system calls like bundle exec rails assets:precompile. To securely run these procedures, we have decided to expose another internal API through a private channel.

This channel is always between the voca backend and the specific decidim instance.

Why did we choose gRPC?

gRPC technology has some advantages that are significant for us:

It handles field renaming, allowing us to version the gateway through the decidim versions more reliably.
It generates client code from gRPC prototypes: since voca wants to use other tools alongside decidim, this is an added benefit as it eliminates the need to rely on programming languages. For example, we could use Java to create an RPC gateway to Metabase, and the strapi backend would handle connections in the same way.
Calls are fast and can be used to stream files.

Prototypes

We use several prototypes for decidim, which will be managed by our voca gem.

    // region/Settings
    rpc GetSettings(google.protobuf.Empty) returns (GetSettingsResponse) { }
    rpc SetSettings(SetSettingsRequest) returns (google.protobuf.Empty) { }

    // region/Seed
    rpc CompileAssets(google.protobuf.Empty) returns (google.protobuf.Empty) {}
    rpc SetupDb(google.protobuf.Empty) returns (google.protobuf.Empty) {}
    rpc SeedAdmin(SeedAdminRequest) returns (SeedAdminResponse) {}

Ping: return “pong” if the message made the roundtrip .
GetSettings: Retrieve all the organization's settings.
SetSettings: Modify any settings for the organization.
CompileAssets: Perform asset compilation.
SetupDb: Perform a db:migrate.
SeedAdmin: Create a general administrator and a /system administrator.

Infrastructure

On every Decidim environment we create on Jelastic, we add a Docker container that will run a gRPC server. This gRPC server (powered by gruf) has the Rails and Decidim context and will be able to look directly at the data models and Decidim code actions. Being on a separate container allows us to have a private gRPC server (mounted on a private network).

RPC In Strapi Backend

Strapi Service `api::decidim.rpc-start-client`

Backend side, we use extensively a service called api::decidim.rpc-start-client that can start a RPC client from an environment data model.

// Exemple of strapi script
const environment = await strapi.entityService.findOne(
	"api::decidim.environment",
  1,
  {populate: {environment_nodes: true}}
)
const rpc = await strapi.service<RPCStartClientCommand>(
    "api::decidim.rpc-start-client",
    {environment}
);
await rpc.query('Ping', {}, {timeout: 8, maxRetry: 1});

strapi.service("api::decidim.rpc-start-client").call(props)

props

prop name	description
`environment`	the strapi `api::decidim.environment` model
`logger`	a function `(msg: string)=>void` that will be used to log information. Default to `console.log`

returns

RPCQueryType instance bound to the first node in nodeGroup=rpc for the environment. During deployment we get all the node of environment and save it in database (see backend/src/jobs/infra/park/pull-nodes.ts). The binding is thus quiet fast.

RPCQueryType.query(methodName, params, options)

props

prop name	description
`methodName`	rpc method name, in camelCase. exemple: `GetSettings`.
`params`	parameters to send to RPC. Should match RPC client types (ex: use enum value like `SETTINGS_SMTP_AUTHENTICATION_CRAM_MD5`)
`options.maxRetry`	number of retry we do the same query. Will retry on any RPC error. Default: 7
`options.timeout`	timeout in second before the RPC to die. use `Infinity` if you don’t want to set a timeout. Default: 5

returns

{data: <RPC response>, code: string | "OK"} The RPC response (promise).

Update and compile prototypes

Prototypes are defined in the monorepo under the contrib/rpc-protos. You can update .proto files in this directory and run then the ./sync shell script to recompile and update the generated client.

Once updated successfully, you will have to update the voca ruby gem manually as the success message instructs:

./sync
# Prototypes updated. Please:
# - update the repository https://github.com/octree-gva/voca-tasks.git
#      1. contrib/rpc-protos/clients/decidim/ruby/decidim_pb.rb -> lib/decidim/voca/rpc/decidim_pb.rb
#      2. contrib/rpc-protos/clients/decidim/ruby/decidim_services_pb.rb -> lib/decidim/voca/rpc/decidim_services_pb.rb

Before updating .proto files, be sure to read the RPC documentation: https://protobuf.dev/programming-guides/proto3/ and the gRPC getting started: https://grpc.io/docs/what-is-grpc/introduction/

RPC In Voca Ruby Gem

# Gemfile
gem "decidim-voca",
		git: 'https://git.octree.ch/decidim/vocacity/tasks.git", 
		branch: 'release/0.26-stable'

Voca ruby gem add a gruf dependancy, that will enable the container user to run a RPC server with bundle exec gruf. The RPC server will load all the rails application, to enable to interact with existing code. So in this RPC server, you will have access to all the classes of decidim.

Here some relevant paths:

lib/decidim/voca/rpc contains the generated code for RPC server (See Update and compile protos section)
app/rpc/decidim/voca/decidim_service_controller.rb contains the rails controller that will route all the methods defined in the proto file.
app/rpc/decidim/voca/rpc are the services called by the controller
app/commands/decidim/voca/organization are the command used by the services.

Sometimes in the voca ruby gem, we will need to execute system calls. To do this, we do call the ruby method system(...) by default, in order to have less injection surface.

# exemple of command that run system call together
# with ruby code, to update locales correctly.
def after_updating_languages
    # Rebuild locales
    system("bundle exec rails decidim:locales:sync_all")
    # Rebuild search tree
    system("bundle exec rails decidim:locales:rebuild_search")
    org = organization.reload
    # Update user locales (they might not have a locale that is now supported)
    ::Decidim::User.where.not(locale: org.available_locales).each do |user|
      user.update(locale: org.default_locale)
    end
  end

info

Internal API for decidim: gRPC​

Why did we choose gRPC?​

Prototypes​

Infrastructure​

RPC In Strapi Backend

Strapi Service api::decidim.rpc-start-client​

Update and compile prototypes​