Database Lab development roadmap


We work hard to develop new features for Database Lab Engine and Platform. Below you can find the main ideas we are working on now or planning to work soon.

Edited: 2020-11-02

Physical provisioning#

Physical provisioning: native support of provisioning from archives created by a specific backup solution or based on an existing Postgres database

  • Support various sources
    • Generic (anything: pg_basebackup, rsync, backup tools)
    • Native support
      • WAL-E/WAL-G backups
      • pgBackRest backups
      • Barman backups
      • pg_probackup backups
      • Nutanix Era
  • Continuously updated state (physical replication based on WAL shipping)
  • Snapshot management (schedule, retention policy)
  • faster WAL replay (pg_prefaulter)

Logical provisioning#

Logical provisioning: native support of provisioning for managed PostgreSQL databases

  • Support various sources
    • Simple dump/restore
    • Amazon RDS
    • Heroku Postgres
    • Azure PostgreSQL
    • Google Cloud SQL for Postgres
    • Digital Ocean Postgres
  • Continuously updated state (logical replication)
  • Snapshot management (schedule, retention policy)
  • Simplified full refresh
  • Partial data retrieval
    • specific tables
    • arbitrary filtering (columns, rows)

Advanced engine features#

  • Persist clones when the engine restarts
  • Point-in-time recovery (PITR) (Can be used for ultra-fast recovery of accidentally deleted data)
  • Duplicate DLE (create a new DLE based on existing one)
  • Utilization alerts
  • Clone analytics
  • User quotas
  • Audit
  • "Tmp" system- and Postgres-level monitoring for clones/sessions
  • Utilization of DLE instance and alerts
  • Usage and estimated savings reports
  • SSH port forwarding for API and Postgres connections
  • Tags
  • Framework for macro database experiments (work with thick clones)
  • Auto-register DLE in Platform

Automated verification of database schema and complex data changes (migrations)#

  • [a] History and logging for clones/sessions
  • [a] Automated detection of locking issues
  • [a] Setting custom statement_timeout
  • [a] PostgreSQL logs for the migration
  • [a] Report in CI and Platform
  • Integration with CI tools – advanced integration
    • GitHub Actions
    • CitcleCI
    • Jenkins
    • GitLab CI/CD
    • Bamboo
    • TravisCI
  • Database migration tools – advanced integration
    • Sqitch
    • Flyway
    • Liquibase
    • Ruby on Rails Active Record
    • Django migrations

Cloning (CoW technology)#

  • ZFS
  • LVM
  • PureStorage
  • Remote clones – Amazon Aurora

Automation, clouds, Kubernetes#

  • Clouds, automation of installation in clients' accounts
    • Basic Terraform templates
    • One-click setup on AWS. AWS Marketplace
    • One-click setup on GCP. GCP Marketplace
    • One-click setup on Azure. GCP Marketplace
    • One-click setup on Alibaba
  • SaaS: cloud offering (fully managed Database Lab)
    • AWS
    • GCP
    • Azure
    • Ali
  • AWS Spot instances for further savings
  • Kubernetes support

SQL optimization – Joe bot#

  • Web UI version
  • Slack chatbot
  • Telegram chatbot
  • History with Search and Share options
  • Visualizations
    • explain.depesz
    • explain.dalibo (PEV2)
    • pgMustard (EE only)
    • FlameGraphs
  • Better optimization recommendations
  • Macroanalysis insights (suggestions based on postgres-checkup / pgss)
  • Hypothetical indexes
  • Hypothetical partitioning
  • Index advisor
  • Utilization control

Data masking and obfuscation#

Better documentation#

  • Tutorials
    • Basic
    • RDS
    • Specific cases
  • User Guides
    • DLE setup and administration
    • Cloning
    • SQL optimization with Joe bot
    • Checkups
    • Building non-production environments
  • References
    • DLE API
    • CLI
    • DLE configuration
    • Joe configuration
  • Explanations and concepts
    • SQL optimization workflow with Database Lab
    • Configuration details
    • Why and how SQL optimization is possible on thin clones
    • Security aspects
    • Secure and robust test/staging environments
  • Video demonstrations