GCP News - 2026-09-05
2026-09-05
最終更新: 2026-09-06 19:18:10 JST
AI による概要
この日は移植と性能検証という、実務寄りの記事が揃いました。Google の財務エンジニアリングチームが、レガシーなデータ層を Spanner へ移行する際に、本番サービスを止めずに二重書き込みを実装するという難題へ取り組んだ事例が公開されています。金額の正確さが要求される高スループットのサービスでは単純な切り替えスクリプトは通用せず、履歴データのバックフィルと検証が終わるまで、レガシーのデータストアと Spanner の双方が同一の書き込みを同時に受けていることを確かめ続ける必要があります。数十個のデータアクセスオブジェクト (DAO) すべてに多段階の二重書き込みを手で書くのは遅く、人的ミスも起きやすいため、チームは Antigravity CLI をヘッドレスモードで動かす自動リファクタリングのパイプラインを構築し、ステージング環境で厳密なデータ同一性を保ちながら移行速度を大きく引き上げました。性能検証では、TPU v6e 上の Gemma 3 12B と 27B を、分類 (プリフィル中心) と生成 (デコード中心) という構造の異なるワークロードで比較したベンチマークが公開されています。生成では 27B が同時 64 ユーザーを超えると明確な性能の壁にぶつかり、128 ユーザーで正規化スループットが 4.12 倍に頭打ちになる一方、12B は 8.19 倍まで伸びました。高い同時実行の生成が必要なら 12B へ落とすか、27B ではレプリカあたりの同時リクエストを 64 に制限するオートスケーリングを設定すべき、という指針が示されています。分類ではパラメータ数の影響がずっと小さく、12B と 27B がともに 6.0 倍前後の同等のスケーリングに達しました。運用の工夫としては、Yahoo が Managed Service for Apache Spark (旧 Dataproc) で柔軟な VM を活用している例が紹介されています。特定のマシン形状がリージョンの容量制約に当たるとクラスタのプロビジョニングが遅れ、重要なデータパイプラインが止まってしまうため、許容できる VM 形状に順位を付けたリストを定義しておき、リージョン内のゾーンを動的に探索して人手を介さずパイプラインの実行を維持する構成です (リージョン全体で容量を探すには Auto-Zone placement の有効化も必要)。リリースノートでは、Google SecOps SOAR の Release 6.3.99 が全リージョンで利用可能になりました。
主要トピック
移行: Google 財務チームが Antigravity CLI のヘッドレスモードで、数十の DAO への二重書き込み実装を自動リファクタリング
性能: TPU v6e 上の生成タスクで Gemma 3 27B は同時 64 ユーザー超で頭打ち (128 ユーザーで 4.12 倍)、12B は 8.19 倍まで伸長
性能: 分類タスクではパラメータ数の影響が小さく、12B と 27B がともに 6.0 倍前後で同等
容量最適化: Yahoo が Managed Service for Apache Spark で許容 VM 形状のランク付きリストを定義し、容量制約を自動で吸収
セキュリティ運用: Google SecOps SOAR の Release 6.3.99 が全リージョンで利用可能に
Google Cloud Release Notes
September 05, 2026
- Link: https://docs.cloud.google.com/release-notes#September_05_2026
- Published: 2026-09-05 16:00:00
- Fetched: 2026-09-06 19:18:10
Google Cloud Blog
Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation
- Link: https://cloud.google.com/blog/topics/developers-practitioners/not-all-llm-workloads-are-equal-benchmarking-tpu-performance-on-classification-vs-generation/
- Published: 2026-09-05 00:36:00
- Fetched: 2026-09-05 05:18:10
詳細を表示
Moving Large Language Models (LLMs) from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economics. Standard hardware benchmarks often ignore a fundamental reality—not all LLM requests stress the silicon in the same way.
In this post, we dive into a comprehensive benchmarking exercise comparing Gemma 3 12B and Gemma 3 27B on Google Cloud TPU v6e to answer a crucial architectural question: How does TPU infrastructure actually perform when tasked with structurally distinct workloads at scale?
Key Findings and Suggestions
Before diving into the methodology, here are the critical takeaways for architects deploying Gemma 3 on TPU v6e:
The Generation Performance Wall
For decode-heavy generation tasks, the Gemma 3 27B model hits a strict performance wall past 64 concurrent users, plateauing at a 4.12x normalized throughput multiplier at 128 users. In contrast, the 12B model scales up to an 8.19x multiplier.
Suggestion: If your workload requires high-concurrency generation, downsize to the 12B model, or set strict pod-autoscaling limits capping concurrent requests at 64 per replica for the 27B model.
The Classification Parity
For prefill-heavy classification tasks, model parameter size matters significantly less. Both the 12B and 27B models achieve similar peak scaling (around 6.0x to 6.4x normalized throughput at 128 users) without saturating the TPUs.
Suggestion: You can safely deploy larger, more capable models for summarization or classification workflows without paying a throughput penalty. The average --max-num-seqs or --max-model-len should be kept judiciously based on the average user load and average tokens per request, without which there might be request drops.
Designing Around the Wall
Hardware saturation manifests as severe latency spikes and silent request dropouts. To mitigate this, do not rely on standard CPU/Memory scaling triggers. Instead, scale based on End-to-End (E2E) latency metrics, and implement aggressive vLLM bucket padding optimizations (VLLM_TPU_BUCKET_PADDING_GAP) to conserve memory.
The Architecture Setup
The inference stack can be divided into three core pillars:
1. Infrastructure: GKE & TPU
The foundation of our deployment is a Google Kubernetes Engine (GKE) Autopilot cluster. Connected to this is a single-host TPU v6e node pool configured with a 2x2 chip topology.
2. Software & Tools: vllm
For the serving framework, we leveraged vllm via vllm-project/tpu-inference.
3. Models: Gemma 3 12B and 27B
We evaluated two highly capable open-weights models: Gemma 3 12B and Gemma 3 27B. These models were accessed via HuggingFace.
The Workloads: Classification vs. Generation
Not all LLM requests stress the system equally. We benchmarked two distinct scenarios: Classification and Generation, across 16, 32, 64, and 128 concurrent users:
- Classification (High Input, Low Output): This use case mimics an e-commerce compliance task. The prompt includes large blocks of product rules, item descriptions, and OCR-extracted text. The output is exceptionally small—typically just classifying an item as "Allow" or "Prohibit". Input Sequence Length (ISL) is ~4,000 tokens and Output Sequence Length (OSL) is ~10 tokens.
- Generation (Low/Medium Input, High Output): This use case mimics long-form text generation. The prompt requests a detailed, analytical policy brief on the future of AI in the labor market. The model spends the majority of its time decoding and streaming out hundreds of tokens. Input Sequence Length (ISL) is 500 tokens and Output Sequence Length (OSL) is ~1,000 tokens.
Results and Observations
We measured metrics like Throughput (requests/sec), End-to-End Latency and the results provided some fascinating insights into how parameter size and hardware bandwidth interact. To ensure architectural consistency, every benchmark was executed using the vllm-project/tpu-inference hardware plugin, leveraging a standardized global serving configuration of max-model-len=128000, max-num-batched-tokens=8192, and max-num-seqs=512.
Generation Scaling Divergence
In Generation tasks, both models perform similarly up to 64 concurrent users. However, at 128 concurrent users, the Gemma 3 12B model shows significantly better scaling, achieving an 8.19x normalized throughput multiplier compared to a 4.12x plateau for the Gemma 3 27B model (normalized against the Gemma 3 12B baseline at 16 users). This suggests that the larger 27B model hits memory or compute limits much earlier under high generation loads.
| Concurrent Users | Gemma 3 12B Throughput (req/s) | Gemma 3 27B Throughput (req/s) |
| 16 users | 1.00 x | 1.05 x |
| 32 users | 1.98 x | 1.97 x |
| 64 users | 2.96 x | 4.00 x |
| 128 users | 8.19 x | 4.12 x |
<div class="article-module h-c-page">
<div class="h-c-grid">
<figure class="article-image--large
h-c-grid__col
h-c-grid__col--6 h-c-grid__col--offset-3
">
<img alt="generation_scaling" src="https://storage.googleapis.com/gweb-cloudblog-publish/images/generation_scaling.max-1000x1000.jpg" />
</a>
</figure>
</div>
</div>
- aside_block
- <ListValue: [StructValue([('title', 'Pro Tip → Metrics Inflation at High Concurrency'), ('body', <wagtail.rich_text.RichText object at 0x7f6776b284c0>), ('btn_text', ''), ('href', ''), ('image', None)])]>
Classification Performance Parity
In Classification tasks, there is negligible difference in scaling behavior between the Gemma 3 12B and Gemma 3 27B models. Both models operate efficiently within the hardware's capacity and scale well, reaching peak normalized throughputs of approximately 6.04x to 6.37x at 128 concurrent users (normalized against the Gemma 3 12B baseline at 16 users).
| Concurrent Users | Gemma 3 12B Throughput (req/s) | Gemma 3 27B Throughput (req/s) |
| 16 users | 1.00 x | 0.76x |
| 32 users | 1.18x | 1.53x |
| 64 users | 2.04x | 3.15x |
| 128 users | 6.37x | 6.04x |
<div class="article-module h-c-page">
<div class="h-c-grid">
<figure class="article-image--large
h-c-grid__col
h-c-grid__col--6 h-c-grid__col--offset-3
">
<img alt="classification_perf_table_image2" src="https://storage.googleapis.com/gweb-cloudblog-publish/images/classification_perf_table_image2.max-1000x1000.png" />
</a>
</figure>
</div>
</div>
Latency Threshold Analysis
End-to-End (E2E) latency exhibits different scaling behaviors depending on the model size and task. When using identical serving hyperparameters (--max-num-seqs=512), the Gemma 3 12B model's Classification latency roughly doubles when moving from 32 users to 64 users, indicating resource contention. However, for the larger Gemma 3 27B model, Classification latency remains relatively flat between 32 and 64 users before doubling at the 128-user mark.
| Model | Task | 16 Users | 32 Users | 64 Users | 128 Users |
| Gemma 3 12B | Generation | 1.00x | 1.13x | 1.40x | 1.70x |
| Gemma 3 12B | Classification | 1.00x | 0.99x | 1.79x | 2.90x |
| Gemma 3 27B | Generation | 1.20x | 1.68x | 2.93x | 3.33x |
| Gemma 3 27B | Classification | 1.20x | 1.95x | 1.95x | 3.88x |
<div class="article-module h-c-page">
<div class="h-c-grid">
<figure class="article-image--large
h-c-grid__col
h-c-grid__col--6 h-c-grid__col--offset-3
">
<img alt="latency threshold_image3" src="https://storage.googleapis.com/gweb-cloudblog-publish/images/latency_threshold_image3.max-1000x1000.png" />
</a>
</figure>
</div>
</div>
- aside_block
- <ListValue: [StructValue([('title', 'A Crucial TPU Optimization Technique'), ('body', <wagtail.rich_text.RichText object at 0x7f6776b28520>), ('btn_text', ''), ('href', ''), ('image', None)])]>
Conclusion
Benchmarking Gemma 3 12B and 27B models on Google Cloud TPU v6e architecture reveals that raw parameter count is not the sole predictor of inference performance; rather, the interaction between the serving framework, hardware topology, and workload token ratios dictates efficiency. For generation tasks (low input, high output), the 12B model proves superior at high concurrency, sustaining an 8.19x relative throughput multiplier where the 27B model saturates at 4.12x. Conversely, for prefill-heavy classification tasks, both models perform similarly, allowing organizations to deploy larger models without a severe scaling penalty. Our evaluation also mapped exact hardware saturation thresholds—such as End-to-End latency doubling at 64 users for classification and hitting a cliff at 128 users for generation—enabling precise, data-driven auto-scaling triggers rather than costly over-provisioning. Ultimately, achieving these peak metrics requires aggressive tuning of vllm parameters, such as adjusting batched tokens and configuring TPU-specific bucket padding to prevent compute waste, proving that cost-effective AI infrastructure must strictly align model selection and serving configurations to the unique input/output profiles of production workloads.
Ready to scale your LLM workloads?
Don't let unoptimized infrastructure bottleneck your enterprise AI rollouts. Now that you know how different workload shapes impact hardware saturation, it's time to put these insights into practice:
- Use these benchmarks to right-size your production architecture. Safely leverage the larger Gemma 3 27B for prefill-heavy classification tasks without a throughput penalty, but consider switching to the 12B model to maintain linear scaling for decode-heavy generation at high concurrency.
- Deploy using Google Kubernetes Engine (GKE) with TPU v6e node pools to build a highly scalable, managed AI foundation and dedicated vllm-project/tpu-inference hardware plugin. Alternatively, you can also deploy via Model Garden on Gemini Enterprise Agent Platform or you can spin up TPU VMs for serving Gemma 3 models.
Have you encountered similar performance walls in your own production deployments? Share your scaling strategies, ask questions, and join the discussion in the Google Cloud Community forums.
How Yahoo optimizes resources with flexible VMs in Managed Service for Apache Spark
- Link: https://cloud.google.com/blog/products/data-analytics/how-yahoo-optimizes-apache-spark-with-flexible-vms/
- Published: 2026-09-05 01:00:00
- Fetched: 2026-09-05 02:22:19
詳細を表示
As a global media and technology company connecting hundreds of millions of users to finance, sports, and entertainment platforms, Yahoo operates a massive data infrastructure where analytics workloads must run continuously at high speed. In deadline-driven data environments, relying on fixed virtual machine (VM) configurations creates a brittle system; if a specific machine shape faces a regional capacity constraint, cluster provisioning in Managed Service for Apache Spark (formerly Dataproc) can experience delays and stall critical data pipelines.
Yahoo utilizes flexible VMs in Managed Service for Apache Spark clusters to automatically absorb these resource fluctuations by defining a ranked list of acceptable VM shapes. This allows the system to dynamically search regional zones and maintain pipeline execution without manual intervention. To search for capacity across a region, teams must also enable Auto-Zone placement.
This optimization builds on Yahoo's broader data modernization journey, which involved migrating on-premises Hadoop and big data estates directly to Google Cloud. By transitioning those legacy workloads, the team established a cloud foundation capable of running high-scale batch and streaming analytics with dynamic resource flexibility.
This post provides a technical blueprint for configuring flexible VM instance rankings in Managed Service for Apache Spark to automatically manage capacity constraints and maintain pipeline execution.
Operational trade-offs of static configurations
Configuring clusters with a single, fixed machine type in a specific zone introduces constraints when regional zonal capacity fluctuations occur, potentially impacting cluster provisioning. Rather than manage these capacity variations through custom retry logic or manual intervention, using flexible configurations allows your infrastructure to automatically adapt. By accepting multiple VM shapes and searching across zones in the selected region, flexible configurations help streamline provisioning to better support high-scale analytics workloads.
Rules for configuring flexible clusters
Deploying flexible configurations requires aligning several connected design choices:
-
Enable auto-zone placement: You must pass a region(--region=${REGION}) or an empty zone string (--zone="") so Managed Spark can search for available capacity across the entire region.
-
Maintain core and memory symmetry: If your Managed Spark cluster uses autoscaling, all machine types in your flexible list must share a similar core count and memory size, even if they come from different VM families. A uniform CPU-to-memory ratio across primary and secondary workers prevents performance degradation, as the smallest ratio determines your effective container sizing.
-
Align component properties: Managed Spark calculates system properties based on VM cores and memory. When mixing machine shapes, you may need explicit property overrides to keep YARN and Spark resource allocations aligned with your expected worker behavior.
Two ways flexible VMs support massive workloads
For large-scale data environments, flexible configurations support operations in two ways:
-
Higher cluster creation success: Instead of failing when a preferred VM type is out of stock, Managed Spark selects from a ranked list to keep provisioning moving.
-
Better regional resource use: Auto-zone placement searches the entire region to find capacity, which reduces provisioning friction during high-demand periods.
gcloud example
- code_block
- <ListValue: [StructValue([('code', 'gcloud dataproc clusters create analytics-cluster \\\r\n --region=us-central1 \\\r\n --zone="" \\\r\n --num-workers=10 \\\r\n --master-instance-selection=\'{"machineTypes":["e2-standard-8"],"rank":0}\' \\\r\n --master-instance-selection=\'{"machineTypes":["n2-standard-8"],"rank":1}\' \\\r\n --worker-instance-selection=\'{"machineTypes":["e2-standard-8"],"rank":0}\' \\\r\n --worker-instance-selection=\'{"machineTypes":["n2-standard-8"],"rank":1}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f78f8cf7700>)])]>
API example
You can also build this capacity policy into your automated pipelines or Managed Service for Apache Airflow DAGS using the instanceFlexibilityPolicy field in the ‘Dataproc’ API:
- code_block
- <ListValue: [StructValue([('code', '{\r\n "projectId": "PROJECT_ID",\r\n "clusterName": "analytics-cluster",\r\n "config": {\r\n "gceClusterConfig": {\r\n "zoneUri": ""\r\n },\r\n "secondaryWorkerConfig": {\r\n "numInstances": 8,\r\n "instanceFlexibilityPolicy": {\r\n "instanceSelectionList": [\r\n {\r\n "machineTypes": ["n2-standard-8"],\r\n "rank": 0\r\n },\r\n {\r\n "machineTypes": ["e2-standard-8", "t2d-standard-8"],\r\n "rank": 1\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f78f8cf72b0>)])]>
This API policy achieves the same goal: it establishes your preferred shape, documents valid fallbacks, and lets Managed Spark resolve resource constraints without breaking your automation scripts.
Establishing an infrastructure policy
Managing data at this scale requires standardizing a clear resource policy rather than relying on a single rigid machine type. Your configuration standards should outline:
-
Preferred and fallback VM families for secondary workers.
-
Default auto-zone placement to enable flexible provisioning.
-
Identical core and memory configurations when using autoscaling.
-
Uniform CPU-to-memory ratios across all worker groups to maintain predictable container sizing.
-
Explicit YARN or Spark property overrides to guarantee consistent runtime behavior across different machine lines.
-
Shuffle-safe patterns for Spark workloads running on Spot or highly elastic capacity.
By adopting flexible configurations, you turn infrastructure scarcity into a predictable fallback plan, keeping your critical data pipelines up and running.
Yahoo impact and results
By implementing flexible VMs in Managed Service for Apache Spark, Yahoo successfully reduced cluster provisioning failures by 85% which were caused by regional capacity stockouts. This flexible configuration allows their data infrastructure to automatically handle capacity constraints and successfully provision resources without requiring manual intervention. As a result, Yahoo ensures continuous workload execution and prevents downstream processing delays across their massive data pipelines.
"Managing high-scale data analytics at Yahoo requires resilient, automated infrastructure. Moving to flexible VMs in Managed Service for Apache Spark has transformed our approach; instead of stalling when a specific machine shape faces capacity constraints, our clusters now automatically pivot to our ranked fallback options. This has helped us reduce provisioning failures by 85%, providing the reliability we need to keep our global media platforms running smoothly." - Akshay Jain, Senior Software Developer Engineer, Yahoo!
Strategic benefits of flexible infrastructure
Adopting a flexible compute stack transforms your environment into a dynamic pool of resources that adapts to your operational needs. By moving away from rigid, single-machine type configurations, you ensure that your workloads reliably access the compute they need, regardless of supply fluctuations. This shift not only maximizes workload obtainability and reliability but also facilitates seamless hardware modernization by allowing you to prioritize newer VM generations while maintaining older types as reliable fallback options.
Build your resilient data pipeline
Transitioning to a fluid compute strategy ensures your critical analytics remain operational despite regional resource shifts. Here is how you can begin optimizing your infrastructure today:
-
Audit your workloads: Identify applications tightly coupled to specific VM families or zones and map out viable alternative hardware shapes.
-
Standardize resource policies: Explore the documentation for Managed Spark flexible VMs to establish your preferred and fallback VM families.
-
Align financial strategy: Utilize Flexible Committed Use Discounts (Flex CUDs) to maintain cost predictability when workloads dynamically pivot to alternative machine types.
-
Claim your credits: New customers may be eligible for $300 in credits to try Managed Service for Apache Spark and other Google Cloud products at no cost.
Spanner migrations: Automating dual-write with Antigravity CLI for minimal disruption
- Link: https://cloud.google.com/blog/topics/developers-practitioners/using-antigravity-cli-to-streamline-dual-write-database-migration/
- Published: 2026-09-05 01:00:00
- Fetched: 2026-09-05 02:22:19
詳細を表示
When Google's Finance Engineering team needed to modernize their legacy data layer, they chose Spanner, a globally distributed, strongly consistent, multi-model database with high availability capabilities. But migrating to Spanner without taking production services offline was a daunting engineering challenge: As the internal team responsible for the application, we needed to manually rewrite dual-write logic across dozens of Data Access Objects (DAOs), a process that is slow and prone to human error. Further, doing so without disruption would have required implementing multi-phase dual-write architectures across every DAO in our codebase.
To solve this, we took an alternative approach: We built an automated refactoring pipeline powered by Antigravity CLI in headless mode. This helped us accelerate our migration velocity significantly while maintaining strict data parity in our staging environments as we prepare for production.
The challenge: Anatomy of a dual-write migration
When migrating high-throughput production services where financial accuracy is essential, simple cutover scripts do not work. You must verify that both the legacy datastore and Spanner receive identical writes simultaneously until all the historical data backfills and verifications are complete.
We structured our migration across three distinct phases:
-
Historical backfill: Copying existing historical records to Spanner while maintaining referential integrity.
-
Dual-write / dual-read implementation: Modifying every DAO to write mutations to both the primary store and Cloud Spanner in parallel during the migration window.
-
Automated API verification and parity checking: Intercepting RPC traffic and verifying end-to-end that every write lands with byte-for-byte equivalence across both stores.
<div class="article-module h-c-page">
<div class="h-c-grid">
<figure class="article-image--large
h-c-grid__col
h-c-grid__col--6 h-c-grid__col--offset-3
">
<img alt="1 - Dual Write Architecture" src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_Dual_Write_Architecture.max-1000x1000.png" />
</a>
</figure>
</div>
</div>
The architectural pattern is clean, but at our scale, we began to encounter friction. That’s because each DAO requires:
-
A dedicated MutationConverter class mapping complex domain models to Spanner schema columns
-
Dual-write branch handling and rollback or error-reporting logic
-
A suite of unit tests verifying both primary and Spanner writes using fake time sources and test doubles (FakeTimeSource)
Performing these identical, high-precision code changes across 30+ DAOs by hand would have taken months of engineering time.
The solution: Standardized mutation converter patterns
To verify that our automation pipeline could reliably generate clean code, we first standardized our DAO refactoring pattern around a decoupled MutationConverter interface.
Instead of embedding raw Spanner table names and column assignments directly inside core DAO business logic, we isolate Spanner schema translation into dedicated converter units:
- code_block
- <ListValue: [StructValue([('code', '// Example of the standardized pattern generated by our pipeline\r\n\r\ntype BpcTransferAmountsMutationConverter interface {\r\n ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n ToUpdateMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n}\r\n\r\ntype bpcTransferAmountsMutationConverterImpl struct {\r\n tableName string\r\n}\r\n\r\nfunc (c *bpcTransferAmountsMutationConverterImpl) ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error) {\r\n if entity == nil {\r\n return nil, errors.New("entity cannot be nil")\r\n }\r\n \r\n // Map domain fields to Cloud Spanner table schema\r\n cols := []string{"TransferId", "AmountCents", "CurrencyCode", "LastModifiedTimestamp"}\r\n vals := []interface{}{\r\n entity.TransferId,\r\n entity.AmountCents,\r\n entity.CurrencyCode,\r\n spanner.CommitTimestamp, // Use Spanner commit timestamps\r\n }\r\n \r\n return spanner.Insert(c.tableName, cols, vals), nil\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f78f90169a0>)])]>
By establishing a rigid, deterministic contract between the DAO and the Spanner SDK (spanner.Mutation), we created an exact target specification that an AI coding agent could reason about and generate reliably.
Why use Antigravity CLI in headless mode?
Interactive AI chat interfaces in IDEs work well for exploratory coding, but they are poorly suited for systematic, multi-file code updates across an entire codebase. When you need to apply repeatable refactoring to dozens of targets without missing edge cases, you need automated workflows.
We addressed this by building an orchestration script (migration_ui.py) that runs Antigravity CLI in headless mode (-p).
Headless mode lets Antigravity run directly inside shell scripts, continuous integration pipelines, and background automation jobs without requiring manual terminal prompts. This approach helped us scale our work in three key ways:
-
Deterministic prompt architectures: We treated our prompts as version-controlled engineering artifacts. We codified precise rules handling common Spanner edge cases — such as timestamp serialization, nullability conversions, mutation ambiguity, and FakeTimeSource test injection — directly into reusable prompt templates.
-
Batch execution and automated verification: Our orchestration script takes a target DAO name as input, retrieves the existing single-write source code and schema, and feeds it to headless Antigravity alongside our structural conventions. Antigravity generates the new converter, the refactored dual-write DAO, and corresponding unit tests. The script then runs blaze test. If a linter error or test assertion fails, the error log feeds directly back into Antigravity for self-correction.
-
Overnight execution at scale: Because the loop runs unattended, engineers can queue up 10 DAOs at the end of the day. By morning, the pipeline generates, tests, and validates 10 clean changelists ready for human code review.
Results and key takeaways for cloud engineers
Combining Spanner's distributed database primitives with Antigravity CLI's headless automation produced clear benefits across our engineering organization:
-
Significant reduction in migration effort: DAO dual-write migrations that previously required extensive manual coding and testing were completed and reviewed in a fraction of the time
-
Highly reliable data migration: Because every generated DAO adhered to the exact same tested MutationConverter pattern and underwent automated unit testing against Spanner test doubles, we sustained high data fidelity during our extensive migration testing.
-
Focus on higher-value engineering: Engineers avoided repetitive boilerplate refactoring, giving them time to focus on data modeling, architectural resilience, and performance optimization.
Three tips for your next database migration
-
Decouple schema translation first: Before writing migration scripts, define a strict interface (like our MutationConverter) that isolates your new cloud database SDK requirements from your existing business logic. AI agents work best when given clear, bounded design patterns.
-
Move from interactive chat to headless automation: When executing repetitive refactoring across more than three or four files, invest in scripted, headless workflows. Treating prompt inputs and test verifications as automated build steps help maintain quality and consistency.
-
Let the build system act as your guardrail: Connect your AI generation loop directly to your build and test harness (bazel test or go test). This lets the model fix compile and assertion errors before a developer reviews the code.
Get started
Whether you’re migrating financial systems or building cloud-native applications from scratch, Spanner and Antigravity provide a foundation for scalable software development.
-
Explore Cloud Spanner: Learn more about Spanner's distributed architecture Google Cloud Spanner documentation.
-
Discover Gemini for Developers: See how AI-assisted coding and headless CLI automation can assist your engineering workflows at Google Cloud AI for Developers.
Google Cloud Blog (AI & ML)
Spanner migrations: Automating dual-write with Antigravity CLI for minimal disruption
- Link: https://cloud.google.com/blog/topics/developers-practitioners/using-antigravity-cli-to-streamline-dual-write-database-migration/
- Published: 2026-09-05 01:00:00
- Fetched: 2026-09-05 02:22:23
詳細を表示
When Google's Finance Engineering team needed to modernize their legacy data layer, they chose Spanner, a globally distributed, strongly consistent, multi-model database with high availability capabilities. But migrating to Spanner without taking production services offline was a daunting engineering challenge: As the internal team responsible for the application, we needed to manually rewrite dual-write logic across dozens of Data Access Objects (DAOs), a process that is slow and prone to human error. Further, doing so without disruption would have required implementing multi-phase dual-write architectures across every DAO in our codebase.
To solve this, we took an alternative approach: We built an automated refactoring pipeline powered by Antigravity CLI in headless mode. This helped us accelerate our migration velocity significantly while maintaining strict data parity in our staging environments as we prepare for production.
The challenge: Anatomy of a dual-write migration
When migrating high-throughput production services where financial accuracy is essential, simple cutover scripts do not work. You must verify that both the legacy datastore and Spanner receive identical writes simultaneously until all the historical data backfills and verifications are complete.
We structured our migration across three distinct phases:
-
Historical backfill: Copying existing historical records to Spanner while maintaining referential integrity.
-
Dual-write / dual-read implementation: Modifying every DAO to write mutations to both the primary store and Cloud Spanner in parallel during the migration window.
-
Automated API verification and parity checking: Intercepting RPC traffic and verifying end-to-end that every write lands with byte-for-byte equivalence across both stores.
<div class="article-module h-c-page">
<div class="h-c-grid">
<figure class="article-image--large
h-c-grid__col
h-c-grid__col--6 h-c-grid__col--offset-3
">
<img alt="1 - Dual Write Architecture" src="https://storage.googleapis.com/gweb-cloudblog-publish/images/1_-_Dual_Write_Architecture.max-1000x1000.png" />
</a>
</figure>
</div>
</div>
The architectural pattern is clean, but at our scale, we began to encounter friction. That’s because each DAO requires:
-
A dedicated MutationConverter class mapping complex domain models to Spanner schema columns
-
Dual-write branch handling and rollback or error-reporting logic
-
A suite of unit tests verifying both primary and Spanner writes using fake time sources and test doubles (FakeTimeSource)
Performing these identical, high-precision code changes across 30+ DAOs by hand would have taken months of engineering time.
The solution: Standardized mutation converter patterns
To verify that our automation pipeline could reliably generate clean code, we first standardized our DAO refactoring pattern around a decoupled MutationConverter interface.
Instead of embedding raw Spanner table names and column assignments directly inside core DAO business logic, we isolate Spanner schema translation into dedicated converter units:
- code_block
- <ListValue: [StructValue([('code', '// Example of the standardized pattern generated by our pipeline\r\n\r\ntype BpcTransferAmountsMutationConverter interface {\r\n ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n ToUpdateMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n}\r\n\r\ntype bpcTransferAmountsMutationConverterImpl struct {\r\n tableName string\r\n}\r\n\r\nfunc (c *bpcTransferAmountsMutationConverterImpl) ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error) {\r\n if entity == nil {\r\n return nil, errors.New("entity cannot be nil")\r\n }\r\n \r\n // Map domain fields to Cloud Spanner table schema\r\n cols := []string{"TransferId", "AmountCents", "CurrencyCode", "LastModifiedTimestamp"}\r\n vals := []interface{}{\r\n entity.TransferId,\r\n entity.AmountCents,\r\n entity.CurrencyCode,\r\n spanner.CommitTimestamp, // Use Spanner commit timestamps\r\n }\r\n \r\n return spanner.Insert(c.tableName, cols, vals), nil\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f52900e51c0>)])]>
By establishing a rigid, deterministic contract between the DAO and the Spanner SDK (spanner.Mutation), we created an exact target specification that an AI coding agent could reason about and generate reliably.
Why use Antigravity CLI in headless mode?
Interactive AI chat interfaces in IDEs work well for exploratory coding, but they are poorly suited for systematic, multi-file code updates across an entire codebase. When you need to apply repeatable refactoring to dozens of targets without missing edge cases, you need automated workflows.
We addressed this by building an orchestration script (migration_ui.py) that runs Antigravity CLI in headless mode (-p).
Headless mode lets Antigravity run directly inside shell scripts, continuous integration pipelines, and background automation jobs without requiring manual terminal prompts. This approach helped us scale our work in three key ways:
-
Deterministic prompt architectures: We treated our prompts as version-controlled engineering artifacts. We codified precise rules handling common Spanner edge cases — such as timestamp serialization, nullability conversions, mutation ambiguity, and FakeTimeSource test injection — directly into reusable prompt templates.
-
Batch execution and automated verification: Our orchestration script takes a target DAO name as input, retrieves the existing single-write source code and schema, and feeds it to headless Antigravity alongside our structural conventions. Antigravity generates the new converter, the refactored dual-write DAO, and corresponding unit tests. The script then runs blaze test. If a linter error or test assertion fails, the error log feeds directly back into Antigravity for self-correction.
-
Overnight execution at scale: Because the loop runs unattended, engineers can queue up 10 DAOs at the end of the day. By morning, the pipeline generates, tests, and validates 10 clean changelists ready for human code review.
Results and key takeaways for cloud engineers
Combining Spanner's distributed database primitives with Antigravity CLI's headless automation produced clear benefits across our engineering organization:
-
Significant reduction in migration effort: DAO dual-write migrations that previously required extensive manual coding and testing were completed and reviewed in a fraction of the time
-
Highly reliable data migration: Because every generated DAO adhered to the exact same tested MutationConverter pattern and underwent automated unit testing against Spanner test doubles, we sustained high data fidelity during our extensive migration testing.
-
Focus on higher-value engineering: Engineers avoided repetitive boilerplate refactoring, giving them time to focus on data modeling, architectural resilience, and performance optimization.
Three tips for your next database migration
-
Decouple schema translation first: Before writing migration scripts, define a strict interface (like our MutationConverter) that isolates your new cloud database SDK requirements from your existing business logic. AI agents work best when given clear, bounded design patterns.
-
Move from interactive chat to headless automation: When executing repetitive refactoring across more than three or four files, invest in scripted, headless workflows. Treating prompt inputs and test verifications as automated build steps help maintain quality and consistency.
-
Let the build system act as your guardrail: Connect your AI generation loop directly to your build and test harness (bazel test or go test). This lets the model fix compile and assertion errors before a developer reviews the code.
Get started
Whether you’re migrating financial systems or building cloud-native applications from scratch, Spanner and Antigravity provide a foundation for scalable software development.
-
Explore Cloud Spanner: Learn more about Spanner's distributed architecture Google Cloud Spanner documentation.
-
Discover Gemini for Developers: See how AI-assisted coding and headless CLI automation can assist your engineering workflows at Google Cloud AI for Developers.