This image shows an architecture diagram for the Databricks Data Intelligence Platform hosted on Google Cloud Platform (GCP).
The image has been marked up by hand to trace the flow of data from its source to its consumption.
Data Flow Traced by the Red Markings
Ingestion Phase: Data starts on the far left under Sources. The handwritten markings group these into semi-structured files/logs, structured RDBMS/Business apps, and external data and image sources
Processing Hub: The arrows show data feeding into ingestion tools like Auto Loader and Pub/Sub Datastream. From there, it flows into Delta Live Tables and processing engines like Spark / Photon.
The "Catalog" Center: A red arrow points directly to a central box labeled "catalog" in handwriting. This highlights Unity Catalog, which manages governance, lineage, and access control for data assets.
Consumption Phase: On the right side, the lines converge on Data Warehousing (Databricks SQL) and Data Analysis. The flow directly toward external AI services like Hugging Face and Vertex AI.
Core Architectural Layers
The architecture is structured vertically to follow the standard data lifecycle:
1. Data Sources & Ingestion
Sources: Captures files, logs, IoT streams, relational databases, and SaaS business applications.
Ingest: Uses Batch & Streaming connectors alongside Google native tools like Cloud Storage Transfer Service and Pub/Sub to move data into the platform.
2. Data Intelligence Platform (The Core)
Storage: Built on top of Google Cloud Storage. It utilizes open formats like Delta Lake and UniForm to keep data accessible.
Governance: Managed entirely by Unity Catalog to ensure secure data sharing and compliance.
Intelligence Engine: Driven by Databricks IQ, which uses built-in AI assistants and predictive optimization to maximize performance.
Processing: Supports both data engineering (via Delta Live Tables) and machine learning workloads (via Mosaic AI).
3. Serving & Analytics
Query & Process: Uses Databricks SQL for high-performance data warehousing queries.
Serve & Analyse: Delivers processed data out to AI models, operational databases (like Cloud Bigtable), and BI dashboards.
這張架構圖展示了部署在 Google Cloud Platform (GCP) 上的 Databricks 資料智慧平台 (Data Intelligence Platform)。
圖中包含手寫紅線標記，用以追蹤資料從源頭到消費端的流向。
紅色標記追蹤的資料流
攝入階段 (Ingestion Phase)：資料始於最左側的資料源 (Sources)。手寫標記將其歸類為半結構化檔案/日誌、結構化 RDBMS/商務應用程式，以及外部資料與影像來源。
處理中心 (Processing Hub)：箭頭顯示資料餵入 Auto Loader 和 Pub/Sub Datastream 等攝入工具。接著，資料流入 Delta Live Tables 以及 Spark / Photon 等處理引擎。
「型錄」中心 (The "Catalog" Center)：紅色箭頭直接指向手寫標記為 "catalog" 的中央區塊。這強調了 Unity Catalog，它負責管理資料資產的治理、歷程追蹤 (Lineage) 與存取控制。
消費階段 (Consumption Phase)：在右側，線條匯聚於資料倉儲 (Databricks SQL) 與資料分析。資料流直接導向 Hugging Face 與 Vertex AI 等外部 AI 服務。
核心架構分層
該架構採垂直分層，遵循標準的資料生命週期：
1. 資料源與攝入 (Data Sources & Ingestion)
資料源 (Sources)：擷取檔案、日誌、物聯網 (IoT) 串流、關聯式資料庫及 SaaS 商務應用程式。
攝入 (Ingest)：利用批次與串流 (Batch & Streaming) 連接器，並結合 Cloud Storage Transfer Service 與 Pub/Sub 等 Google 原生工具，將資料搬移至平台中。
2. 資料智慧平台 (核心層)
儲存 (Storage)：構建於 Google Cloud Storage 之上。採用 Delta Lake 與 UniForm 等開放格式，以確保資料的可存取性。
治理 (Governance)：完全由 Unity Catalog 管理，以確保資料安全共享與合規性。
智慧引擎 (Intelligence Engine)：由 Databricks IQ 驅動，內建 AI 助手與預測優化功能，能大幅提升效能。
處理 (Processing)：同時支援資料工程（透過 Delta Live Tables）與機器學習工作負載（透過 Mosaic AI）。
3. 服務與分析 (Serving & Analytics)
查詢與處理 (Query & Process)：使用 Databricks SQL 進行高效能的資料倉儲查詢。
服務與分析 (Serve & Analyse)：將處理後的資料傳遞至 AI 模型、維運資料庫（如 Cloud Bigtable）以及 BI 儀表板。