Skip to main content

DLE 2.5: Better data extraction for logical mode and configuration improvements

· 4 min read

DLE 2.5: Better data extraction for logical mode and configuration improvements

note

Action required to migrate from a previous version. See Migration notes.

The Database Lab Engine (DLE) is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their own database provisioned in seconds without extra costs.

DLE 2.5 significantly expands the capabilities of automatic preparation of snapshots directly from managed database services, as well as from logical dumps, namely:

  • restoring of multiple databases
  • various pg_dump output formats and file compression formats

Since version 2.5, it becomes possible to reset the clone's database state to a specific snapshot if multiple snapshots are available. See DLE CLI reference. There is also a new option for the reset command, --latest, that allows resetting to the latest available state not knowing the snapshot name. This can be very useful in situations when a clone lives long, occupying a specific port, and some applications (e.g., analytical tools) are configured to work with it – users can periodically switch to the freshest database state without a need to reconfigure their applications.

All new restore features are also already available in the Terraform module (currently works with AWS only).

Additionally, this release has a lot of improvements and fixes. Read the full changelog.

Migration notes​

The Database Lab ecosystem is growing with new products and services. Most of them have configuration files and metadata, but their structure and naming are not uniform.

To simplify work with the configuration, we decided to redesign the configuration structure of all Database Lab services and standardize the naming of configuration files and directories.

Prepare configuration directories on a host machine​

  • The main directory of Database Lab configuration is ~/.dblab. There are directories for each service inside this main directory:
    • cli - configs of Database Lab command-line interface
    • engine - configs and metadata of Database Lab Engine
    • joe - configs and metadata of Joe Bot assistant
    • ci_checker - configs of DB Migration Checker
  • Rename the configuration file of Database Lab Engine to server.yml and store it to ~/.dblab/engine/configs
  • In the running command docker run ... replace the mounting flag
    --volume ~/.dblab/server.yml:/home/dblab/configs/config.yml
    with
    --volume ~/.dblab/engine/configs:/home/dblab/configs \
    --volume ~/.dblab/engine/meta:/home/dblab/meta

Check the example of configuration structure on a host machine:

 ~/.dblab/
    cli/
     - cli.yml

    engine/
      configs/
        - server.yml
      meta/
        - state.json

    joe/
      configs/
        - joe.yml
      meta/
        - sessions.json

    ci_checker/
      configs/
        - ci_checker.yml

(optional) Compact the configuration file using common YAML-sections and anchors​

Database Lab Engine starts a number of Docker containers with a PostgreSQL instance inside. A powerful configuration system allows controlling various properties of running containers (PostgreSQL version, Docker image, container parameters, extensions loaded, PostgreSQL configuration parameters).

However, such flexibility may be inconvenient in the case of a number of the configuration section. On the other hand, YAML anchors and aliases can be used, help you conveniently manage your configuration sections

YAML allows defining a "global" binding with & where you can set properties for all PostgreSQL containers and then refer to it using an alias denoted by *.

⚠ Note, the "local" configuration of each section would still be supported, overriding particular parts.

There is an example using an anchor:

# Configure database containers
databaseContainer: &db_container
  dockerImage: 'postgresai/extended-postgres:13'
  containerConfig:
    'shm-size': 1gb

retrieval:
  jobs:
    - logicalDump
    - logicalRestore
    - logicalSnapshot

  spec:
    logicalDump:
      options:
        <<: *db_container
        dumpLocation: '/var/lib/dblab/dblab_pool/dump'

See more examples of configuration files in the Database Lab repository

If you have problems or questions, please contact our communities for help: https://postgres.ai/docs/questions-and-answers#where-to-get-help

Request for feedback and contributions​

Feedback and contributions would be greatly appreciated:

Share this blog post:

Nikolay Samokhvalov
Nikolay Samokhvalov

CEO & Founder of Postgres.ai

Working on tools to balance Dev with Ops in DevOps

Database Lab
Database Lab by Postgres.ai

An open-source experimentation platform for PostgreSQL databases. Instantly create full-size clones of your production database and use them to test your database migrations, optimize SQL, or deploy full-size staging apps.