In the traditional architecture of modern data engineering, organizations frequently fall into a pattern of data duplication. A standard configuration might deploy Google Cloud’s BigQuery for business analytics, Apache Spark for heavy-duty data pipelines, and Trino for the ad-hoc analytical queries that land late on a Friday afternoon. To make this multi-engine setup function, each distinct execution engine maintains its own separate copy of the underlying data. Background synchronization jobs are then tasked with keeping these various copies aligned, a process that is as resource-intensive as it is fragile.
Despite the best efforts of data engineering teams, maintaining absolute consistency across multiple data copies is notoriously difficult, and the inevitable lag between synchronization cycles frequently creates operational friction. Imagine a common business scenario where BigQuery reports that a customer’s order status is marked as shipped, while the Spark data copy simultaneously indicates that the same order is paid and awaiting shipment. Business stakeholders cannot reliably trust either dashboard until the next scheduled synchronization job finally finishes running. Even if the sync succeeds, the organization is paying twice over: once for the continuous compute time required to perform the synchronization, and again for the redundant storage consumed by multiple data copies. If a synchronization job fails, engineers must spend valuable time tracing and resolving pipeline errors.
To address these inefficiencies, Google Cloud’s borderless Lakehouse and the Apache Iceberg open table format have emerged as a powerful combination. This architecture allows multiple analytical and processing engines to read the exact same Parquet files and metadata layout without requiring separate copies for each system. A shared catalog and a rigorous commit protocol ensure that all engines agree on the precise location of the current table version and how changes are committed without one writer overwriting another.
To evaluate how this architecture handles real-world workloads, independent implementations have demonstrated that a single orders table, hosted in a single Cloud Storage bucket and registered within the Lakehouse runtime catalog, can successfully support multiple engines simultaneously. In such setups, Apache Spark creates and loads the table, Trino appends new records, and BigQuery updates and merges rows in place. Testing concurrent writes by commanding Spark and Trino to write to the table at the exact same moment provides clear insight into how Iceberg manages write collisions.

The underlying mental model of this architecture is streamlined and efficient. Every data file and metadata file associated with a table resides in a single Cloud Storage location. The catalog itself does not store any actual data records. Instead, it holds a single piece of critical information: a pointer to the table’s current version via its metadata.json file. By changing this pointer, the system effectively updates the table.
Both Spark and Trino access the Lakehouse runtime catalog through the standard Apache Iceberg REST catalog endpoint, while BigQuery integrates seamlessly without requiring any separate connector setup, reading the table directly through a structured four-part naming convention known as the P.C.N.T. naming structure, which parallels standard BigQuery dataset syntax. Organizations can choose between different credential modes for bucket access, such as credential vending mode, where only the catalog’s dedicated service account needs direct access to the storage bucket, allowing administrators to manage permissions securely through catalog roles.
The initialization of this architecture typically begins with the creation of the storage bucket followed by the registration of the Iceberg catalog using cloud command-line interfaces. By configuring the catalog type as a multiple-bucket catalog, administrators gain the flexibility to expand storage paths over time. The catalog is assigned its own dedicated service account, ensuring that underlying data files remain secure while computational engines interact exclusively through the managed catalog interface.
With the foundational storage and catalog infrastructure established, Apache Spark is generally utilized to create the initial table schema and load baseline data. Establishing the table’s write settings from the very first commit is crucial, as modifying write configurations after multiple engines have begun writing can lead to incompatible file layouts. Spark SQL commands define the table properties, designating copy-on-write as the operational mode for updates, deletes, and merges. Copy-on-write ensures that data manipulation language operations and automated table maintenance tasks rewrite the affected data files directly rather than generating separate delete files alongside them.
Once the table is initialized and populated with initial records, Trino can be introduced to handle read operations and data appends. In a well-designed architecture, responsibilities are logically divided among engines. Because open-source Trino connectors typically write row-level changes using merge-on-read techniques, whereas automated table maintenance and certain DML operations require copy-on-write, row-level mutations are best handled by Spark and BigQuery, leaving Trino to focus efficiently on inserts and selections. Connected through the Iceberg REST catalog, Trino can immediately query and append new records to the shared table without administrative overhead.

BigQuery integrates into this multi-engine ecosystem with equal fluidity, reading the shared table directly through its four-part name without requiring external table definitions or dedicated connection resources. When BigQuery executes updates or complex merge statements to modify order statuses or upsert new records, the changes are committed directly to the shared Iceberg table. Because the table is updated natively, there are no synchronization delays; subsequent queries executed through Trino instantly reflect the updated states across all records.
The true validation of this multi-engine architecture lies in its snapshot history, which explicitly records operations executed by Spark, Trino, and BigQuery within a single timeline. Iceberg relies on an optimistic concurrency control model to manage these interactions. Each writer reads the current table metadata, writes its new data files and a corresponding metadata file independently, and then requests the catalog to update the table pointer from the original version to the newly generated one. If another write operation has committed a change in the interim, the catalog refuses the pointer swap, prompting the writer to refresh, evaluate whether its changes remain valid against the new state, and retry the commit.
In cloud environments configured with Lakehouse runtime catalogs, strict conflict detection is applied when DML operations or automated table maintenance are enabled. Any concurrent modification to the metadata file causes competing commits to undergo serialization per table, which guarantees snapshot isolation for read operations. When concurrent workloads are deliberately forced—such as running a long-running Spark merge operation while simultaneously executing Trino insert statements—the system behaves predictably. Quick-running operations like Trino appends typically reach the catalog first, causing slower analytical operations to encounter validation exceptions if their broader scans overlap with newly introduced data files.
Configuring appropriate isolation levels allows engineering teams to balance consistency and concurrency depending on specific business requirements. While serializable isolation protects operations from missing newly introduced data, snapshot isolation enables multiple non-conflicting writes to commit successfully by building subsequent changes on top of intermediate versions. Throughout these concurrent operations, read-only analytical queries executed via BigQuery remain entirely stable, consistently returning data that aligns precisely with committed snapshots without ever exposing intermediate, half-written states to users.
Ultimately, uniting multiple computational engines around a single Iceberg table, a single storage bucket, and a single catalog eliminates the operational burden of maintaining complex synchronization pipelines. Spark, Trino, and BigQuery can collaboratively read, write, and update the same underlying dataset while ensuring that every engine maintains an accurate, real-time view of the enterprise data. By carefully selecting isolation levels, managing commit retries, and understanding the operational characteristics of each execution engine, organizations can deploy unified lakehouse architectures that deliver robust performance without the traditional costs of data duplication.

