Skip to main content

How does Voca deploy a new Decidim?

Voca needs to deploy a decidim in less than 2minutes. But the deployment process takes up to 16min in the most complicated case. Voca reduces the time of installation by preparing a set of empty decidim instance, we call this parking.

So Voca uses a parking system to always have available instances. Once a user want a new decidim, he creates an instance, that will be paired to parked environment. An instance keeps user’s settings, a parked environment keeps informations about the Jelastic infrastructure.

What do we park?​

A standard infrastructure, ready for scale

Even for the most basic installation, we are ready to scale, horizontally and vertically. To do this, we setup Docker containers in jelastic to have:

  • A redis database
  • A memcached database
  • A nginx
  • A decidim server, with our special sauce
  • A internal decidim API server.
  • A safe storage node

Decidim docker image for Voca

We leverage a docker image for voca usage. This includes:

  • specific entrypoint and hooks for jelastic infrastructures, to ensure UNIX permissions are right (shared volume storage can have have some problems, depending on starting order)
  • a gem voca with:
    • a new process, gruf that exposes an internal Decidim API through gRPC protocol.
    • a middleware that will allow to run decidim with dynamic configuration (normally currency is setted through environnement variables). So here we can change some values dynamically.

Voca DB models: Environment and instance​

In the Strapi backend, we define two important models:

  • api::decidim.environment stores information about the Jelastic environment. You will find attributes like:
    • envName The uniq name of the Jelastic environment, use to know where the environment is installed.
    • subdomain The subdomain used for routing
  • api::decidim.instance stores the links between environments, workspaces, creators, and some non-Decidim data like:
    • smtp_credentials the credentials the users wants to use in their decidim
    • custom_domain The URL decidim will use
    • subdomain the default subdomain of the instance
    • isPaused if the user wants to pause it’s instance
    • …. and all the others settings are directly sync with the decidim instance through RPC

Parking process​

Every time a Voca user wants to create a Decidim instance, we check the number of parked instances. If the number is below a limit (defined as MAX_PARKED_ENV environment value), we enqueue as many parking processes as needed.

When we tell jelastic to install a new parked instance, we send to jelastic a manifest containing infrastructure definition, and some scripts:

  1. Setup the nodes
    • cp: The Decidim images with the supervisord process manager that will start puma, sidekiq, nginx, and a gruf process.
    • sql: A Postgres image, always in alpine. The credentials are defined in environment values.
    • nosql: A Redis image, always in alpine, running in AOF (Append Only Mode). The credentials are set up in the CMD command. This is done to enqueue jobs.
    • cache: A Memcache image to handle Decidim cache.
    • storage: A jelastic/storage image optimized to store data. (see Volumes Mapping section).
  2. Populate volumes
    1. Transfert to the storage nodes the directories needed by decidim
      1. public folder
      2. config folder
      3. storage folder
      4. log folder
    2. Check that the decidim rpc and cp have the same volumes than storage during the install process.
  3. Prepare instance
    1. Run migrations
    2. Install additional modules (decidim awesome, decidim term customizer).
    3. Compile assets once.
  4. Backups and Log observability
    • Set up restic for backup management in the sql and storage groups. The Voca-Decidim images already have restic installed.
    • Link log files for Jelastic to be able to display them in the log sidebar.
  5. We are done.

This installation manifest will also install a hook on before stop, to be sure the instance goes in stop mode with:

  • The assets are precompiled
  • The migrations are done
  • The UNIX permissions of shared folders are right

Volumes Mapping​

We use a storage node to store all the data used by other containers of the environment, to let containers be stateless. All the environment’s volumes defines storage node as volume destination.

See parking process in action​

To see and understand the parking process, you can:

  1. Access your strapi administration http://<admin>/admin
  2. Create a new environment
  3. Access your jobs dashboard http://<admin>/_jobs . Credentials are defined as environments: BULLMQ_USERNAME and BULLMQ_PASSWORD
  4. See your Jelastic dashboard in action

Jelastic Manifests

We use several Jelastic Manifest (https://docs.cloudscripting.com) to run deployment and other actions. All the jelastic manifests are present in the voca-system mono-repo. They are located in the contrib/jelastic-manifests/scripts directory.

  • park.yml Manifest to deploy a new decidim and park it.
  • control.yml Start or stop an environment
  • restart.yml Restart decidim application nodes (cp and rpc nodeGroup).

Debug a manifest script​

To debug or test your manifest script, you have two ways:

  1. run an import through jelastic UI. better to see actions in processes and debug a problem that happens while the manifest is applied
  2. call the jelasticClient from a strapi console. Better to understand returns values from jelastic.

With an import through Jelastic UI:

From a strapi console

cd $ROOT && PORT=1330 yarn strapi console
# Wait for console to start
await strapi.service("api::decidim.infa").call({envName: `test-${+new Date()}`}).install("https://git.octree.ch/decidim/vocacity/system/-/raw/main/contrib/jelastic-manifests/scripts
/park.yml",{IMAGE_REGISTRY: "git.octree.c
h:4567",IMAGE_PATH: "decidim/vocacity/system/decidim-0.26:latest",IMAGE_USERNAME: "tk",IMAGE_PASSWORD: "my-token-password"})

View the Logs​

Your script can takes several minutes to execute a manifest. You can see the jelastic logs from the Import button, JPS tab, and “View Log” button.

References

Install process (Install)​

The install process is the process of binding a parked instance to a live environment. The process takes inputs from the Voca users and works on an already prepared infrastructure.

Here is the standard process to install a vanilla template:

  1. Pick up a parked environment. If none are available, enqueue parked processes and retry later.
  2. Start the selected environment.
  3. Connect to the Decidim private API
    1. Decidim environment open a rpc server in the nodeGroup=rpc, to expose a private API. The strapi backend will connect to it to be able to communicate.
  4. Seed an administrator with the credentials generated by the Strapi backend.
  5. Define settings like name, locales, currencies, timezones through the SetSettings API call.
  6. Start the server and wait for it to be up.
  7. Configure the Traefik router to the subdomain chosen by the Voca user.
    • Add a .yml file in the Traefik configuration with the correct local IPv4 for the Voca subdomain. The yml file is generated from a template (see contrib/jelastic-manifests/templates/traefik-decidim.yml)
  8. We are done, an email is sent.

Templates: vanilla and non-vanilla​

Templates are decidim instances that have been backuped in order to seed decidim instance. A seed will contains:

  • Custom gems
  • Custom configurations

A vanilla template is a raw-decidim, with a fresh install and nothing but the essentials modules installed: decidim_awesome, decidim_term_customizer and decidim.

Install non-vanilla templates​

The process to install a non-vanilla templat****e:

  1. Pick up a parked environment. If there is none available, enqueue parked processes, and retry later.
  2. Start the selected environment
  3. Restore the backup from the template (database, files, and configurations)
  4. Connect to through the rpc channel exposed by the cp's gruf process.
  5. [… same process as vanilla-install]

info