Amazon Web Services (AWS) has announced a significant evolution for its flagship relational database service, Amazon Aurora PostgreSQL, introducing a powerful new capability that allows users to query operational data and data lake assets simultaneously. By integrating DuckDB technology directly into the database engine, AWS is effectively removing the long-standing architectural friction associated with moving, duplicating, and synchronizing data between operational systems and analytical data lakes.
For years, developers and data engineers have faced a difficult trade-off when designing data-heavy applications. Operational data—the live, transactional records powering applications—typically resides in a high-performance relational database like Aurora. Meanwhile, historical data, archives, and bulk analytical datasets are stored in low-cost, scalable data lakes using formats such as Apache Iceberg and Apache Parquet on Amazon S3. To bridge these two worlds, organizations have historically relied on complex "reverse ETL" (Extract, Transform, Load) pipelines. These pipelines not only introduce significant infrastructure costs and engineering overhead but also create latency gaps, as data must be periodically moved and reconciled to provide a unified view.
With this latest update, available on Aurora PostgreSQL versions 17 (starting with 17.11) and 18 (starting with 18.6), that necessity is being fundamentally challenged. Developers can now use their existing PostgreSQL applications and tools to query data across both operational tables and external data lake formats without moving or duplicating a single byte.
Integrating DuckDB for High-Performance Analytics
The technical core of this new functionality is the integration of DuckDB, the popular, high-performance analytical query engine, directly into the Aurora PostgreSQL service. Following the acquisition of the DuckLabs team—the creators behind the open-source DuckDB project—AWS has moved quickly to leverage that expertise to improve the analytical capabilities of its managed services. By embedding the engine within Aurora, AWS has ensured that query processing remains local to the database. There are no additional network hops or external middleware layers required; the database engine itself handles the heavy lifting of reading and interpreting data lake files.
This integration is particularly impactful for modern application development, especially as organizations increasingly rely on AI agents. These agents often require access to vast amounts of context—ranging from live, uncommitted transactional writes to years of archived historical data—to make accurate decisions. Previously, predicting which specific datasets an AI agent might need and pre-replicating that data was an impractical, if not impossible, task. Now, the application can query both sources in a single, unified SQL statement, providing the AI agent with a comprehensive, real-time context window.
Operational Efficiency and Architectural Simplification
The operational benefits of this change are extensive. Because the system utilizes the familiar PostgreSQL syntax, teams can transition to this new architecture without retraining their developers or replacing their existing toolchains. The process begins by enabling the aurora_analytics extension and attaching an IAM role with the AuroraAnalytics feature to the cluster. This role grants the database the necessary permissions to access Amazon S3 buckets and the AWS Glue Data Catalog, which acts as the central metadata repository for the organization’s data lake.
Once configured, users can define foreign tables that point directly to Iceberg or Parquet files. One of the most user-friendly aspects of this implementation is the schema inference capability. When creating a foreign table, the database can automatically read the metadata from the underlying Parquet file, eliminating the tedious and error-prone requirement of manually defining table columns. For larger environments, this is taken a step further with the IMPORT FOREIGN SCHEMA command, which allows developers to bulk-create foreign tables for an entire database of Iceberg or Parquet files in one go, dramatically accelerating the time-to-value for analytical projects.
Advanced Query Optimization and Scalability
A primary concern when querying data lakes directly from an operational database is performance. AWS has addressed this by ensuring that the Aurora engine applies sophisticated optimizations such as predicate pushdown and column pruning. These techniques ensure that the database reads only the necessary rows and columns from the S3 files, rather than pulling entire datasets into memory. Furthermore, the system includes a caching mechanism for frequently accessed data, which significantly accelerates subsequent queries.
Users can monitor the performance of these operations through the aurora_analytics_stat_statements() function. This diagnostic tool provides deep visibility into the query lifecycle, reporting metrics like the number of rows scanned, the volume of bytes read from Amazon S3, and the effectiveness of the local cache. This level of transparency is vital for teams looking to fine-tune their workloads and maintain high performance as their data lake scales.
The capability also extends to federated environments. By registering external, Iceberg-compatible catalogs with the AWS Glue Data Catalog, organizations can join data across multiple disparate systems in a single query. This creates a virtualized layer where data lives wherever it is most cost-effective—whether in a high-performance Aurora instance or an S3 data lake—while appearing to the application as a single, cohesive source.
Flexibility for High-Latency Requirements
While direct querying is highly efficient for most analytical use cases, AWS acknowledges that some applications require single-digit-millisecond latency that may be difficult to guarantee when reading from a remote data lake. To accommodate these scenarios, the new feature supports the materialization of data lake records directly into native Aurora PostgreSQL tables.
Using standard commands such as CREATE TABLE AS SELECT, INSERT INTO, or MERGE INTO, developers can selectively import "hot" data from the data lake into the operational database. This allows for a hybrid approach: the system can query cold, historical data directly from S3 when needed, while keeping frequently accessed, performance-sensitive data locally within the Aurora instance. Because read queries can be offloaded to any read replica within an Aurora cluster, analytical scans do not necessarily compete with the primary writer instance for resources, ensuring that the operational performance of the database remains stable even under heavy analytical load.
Availability and Getting Started
This functionality is currently available in all commercial AWS Regions as well as AWS GovCloud (US) Regions. A significant highlight for AWS customers is that this capability is offered at no additional charge beyond the standard costs of the resources utilized. Users simply pay for the incremental compute capacity consumed by the analytical queries and the standard S3 request costs associated with reading files from the data lake.
For teams ready to implement this, the process is well-supported by documentation provided in the Amazon Aurora User Guide. Whether through the Amazon RDS console or via a standard PostgreSQL client like psql, the path to integrating data lake assets into operational workflows is now more streamlined than ever. As AWS continues to integrate the efficiency of the DuckDB project into its service offerings, this release marks a foundational step toward a more unified, simplified data architecture for the cloud, where the barriers between transactional and analytical systems are increasingly becoming a thing of the past. Organizations looking to provide feedback or share their experiences with this new feature are encouraged to connect via the AWS re:Post community or their dedicated AWS Support contacts.
