Skip to main content

· 2 min read

Meet Joe​

Update: this post reached the HN top, see the discussion of Joe bot at Hacker News!

Joe is a Postgres query optimization assistant. Joe allows to boost the development process:

  • eliminating annoying waiting time needed to provision copies of large databases for development and testing purposes,
  • helping engineers understand details of SQL query performance.

Joe works on top of Database Lab. Every time when an engineer starts communicating with Joe, a new full-size copy of the database is provisioned.

This process is fully automated and takes only a few seconds, even for multi-terabyte databases. Such database copies are called "thin clones" because multiple clones share the same data blocks, so provisioning is super fast, and disk space consumption is very low. The clones are fully independent, so developers can modify databases. Finally, SQL execution plans are identical to production, which makes it possible to troubleshoot and optimize queries reliably without involving production databases.

Currently, Joe is provided only in the form of Slack chatbot. Slack was chosen to improve the level of collaboration of developers and DBAs. Alternative commucation ways (including beloved psql) are planned for future versions.

More about Joe features you can find in "What Is Joe Bot?".

Demo​

Joe demo

· One min read

Database Lab Engine updated to 0.2: everything in containers, better API and CLI​

Update: see the discussion on Hacker News!

We have released version 0.2 of Database Lab Engine, an open source technology that helps you clone non-production databases in seconds.

Now all its components run in containers, so installation and use is much easier. Additionally, various improvements were made, including those in API and client CLI.

Work on documentation continues: we reworked Tutorial, and added new texts. One of them is What is Database Lab. This picture should help to compare Database Lab to traditional methods of development and testing involing large databases:

Comparison Matrix

· 2 min read

Postgres.ai team is proud to announce the very first public release of Database Lab Engine​

Update: see the discussion on Hacker News!

Database Lab Engine helps you build non-production environments for projects that use multi-terabyte Postgres databases. Initially obtained using standard "thick" copying (such as pg_basebackup, restoration from an archive, or dump/restore), Postgres data directory then gets cloned on request. Such cloning takes just a couple of seconds. Developers, DBAs, and QA engineers can quickly get fully independent copies, perform testing, and idea verification obtaining reliable (close to production) results. As a result, development speed and quality significantly increase.

Database Lab Engine is open source, you can find the code, ongoing work, and the Issue tracker here: https://gitlab.com/postgres-ai/database-lab.

Here is the list of some tasks that Database Lab Engine can help solve:

  1. Troubleshoot an SQL query (run EXPLAIN, EXPLAIN (BUFFERS, ANALYZE)): with query planner settings matching production, one can check any query, including UPDATE, DELETE, INSERT, TRUNCATE, not putting production master into any risks. See also: Joe bot.
  2. Verify an index idea: it is easy to create an index and check if it helps optimize your queries.
  3. Check database migrations (DB schema changes) or massive data modifications and highlight potentially dangerous steps, to avoid performance degradation and downtime on production.

A single Database Lab instance can provide multiple thin Postgres clones (full-size and fully independent) simultaneously. It becomes possible thanks to copy-on-write (CoW) technology. The only option supported in version 0.1 is ZFS; however, there are plans to support other technologies in the future.

Database Lab can be installed either on a physical machine or a VM. Both on-premise or cloud setups are possible. Users communicate with Database Lab using either REST API or client CLI. The first version of Database Lab has certain limitations:

  • it works on Ubuntu 18.04 only,
  • only Postgres versions 9.6, 10, 11, and 12 are supported,
  • in addition to ZFS, the installation of Postgres and Golang is required (it is planned to get rid of this requirement in version 0.2, fully switching to containers).