Database Lab development roadmap

Roadmap#

We work hard to develop new features for Database Lab Engine and Platform. Below you can find the main ideas we are working on now or planning to work soon.

Edited: 2020-11-02

Physical provisioning#

Physical provisioning: native support of provisioning from archives created by a specific backup solution or based on an existing Postgres database

  • Support various sources
    • Generic (anything: pg_basebackup, rsync, backup tools)
    • Native support
      • WAL-E/WAL-G backups
      • pgBackRest backups
      • Barman backups
      • pg_probackup backups
      • Nutanix Era
  • Continuously updated state (physical replication based on WAL shipping)
  • Snapshot management (schedule, retention policy)
  • faster WAL replay (pg_prefaulter)

Logical provisioning#

Logical provisioning: native support of provisioning for managed PostgreSQL databases

  • Support various sources
    • Simple dump/restore
    • Amazon RDS
    • Heroku Postgres
    • Azure PostgreSQL
    • Google Cloud SQL for Postgres
    • Digital Ocean Postgres
  • Continuously updated state (logical replication)
  • Snapshot management (schedule, retention policy)
  • Simplified full refresh
  • Partial data retrieval
    • specific tables
    • arbitrary filtering (columns, rows)

Advanced engine features#

  • Persist clones when the engine restarts
  • Point-in-time recovery (PITR) (Can be used for ultra-fast recovery of accidentally deleted data)
  • Duplicate DLE (create a new DLE based on existing one)
  • Utilization alerts
  • Clone analytics
  • User quotas
  • Audit
  • "Tmp" system- and Postgres-level monitoring for clones/sessions
  • Utilization of DLE instance and alerts
  • Usage and estimated savings reports
  • SSH port forwarding for API and Postgres connections
  • Tags
  • Framework for macro database experiments (work with thick clones)
  • Auto-register DLE in Platform

Automated verification of database schema and complex data changes (migrations)#

  • [a] History and logging for clones/sessions
  • [a] Automated detection of locking issues
  • [a] Setting custom statement_timeout
  • [a] PostgreSQL logs for the migration
  • [a] Report in CI and Platform
  • Integration with CI tools – advanced integration
    • GitHub Actions
    • CitcleCI
    • Jenkins
    • GitLab CI/CD
    • Bamboo
    • TravisCI
  • Database migration tools – advanced integration
    • Sqitch
    • Flyway
    • Liquibase
    • Ruby on Rails Active Record
    • Django migrations

Cloning (CoW technology)#

  • ZFS
  • LVM
  • PureStorage
  • Remote clones – Amazon Aurora

Automation, clouds, Kubernetes#

  • Clouds, automation of installation in clients' accounts
    • Basic Terraform templates
    • One-click setup on AWS. AWS Marketplace
    • One-click setup on GCP. GCP Marketplace
    • One-click setup on Azure. GCP Marketplace
    • One-click setup on Alibaba
  • SaaS: cloud offering (fully managed Database Lab)
    • AWS
    • GCP
    • Azure
    • Ali
  • AWS Spot instances for further savings
  • Kubernetes support

SQL optimization – Joe bot#

  • Web UI version
  • Slack chatbot
  • Telegram chatbot
  • History with Search and Share options
  • Visualizations
    • explain.depesz
    • explain.dalibo (PEV2)
    • pgMustard (EE only)
    • FlameGraphs
  • Better optimization recommendations
  • Macroanalysis insights (suggestions based on postgres-checkup / pgss)
  • Hypothetical indexes
  • Hypothetical partitioning
  • Index advisor
  • Utilization control

Data masking and obfuscation#

Better documentation#

  • Tutorials
    • Basic
    • RDS
    • Specific cases
  • User Guides
    • DLE setup and administration
    • Cloning
    • SQL optimization with Joe bot
    • Checkups
    • Building non-production environments
  • References
    • DLE API
    • CLI
    • DLE configuration
    • Joe configuration
  • Explanations and concepts
    • SQL optimization workflow with Database Lab
    • Configuration details
    • Why and how SQL optimization is possible on thin clones
    • Security aspects
    • Secure and robust test/staging environments
  • Video demonstrations