Introduction & Workspace Basics
Hey, hello everyone. We will start by thinking about the Data Platform Architecture on GCP (Google Cloud Platform).
First of all, the first question is: Where is the storage credential? The answer is the Workspace and Data Explorer.
Here is why: when we first create everything on the cloud server, we need to create the identity credential for the end-user first. That's why we need to store the credential on the Workspace. Using the Data Explorer, we can grant access to the end-user so they can access the Databricks resources deployed on Google Cloud. After that, we need to create the Workspace, which is the predefined resource when we try to use Databricks.
Question: What are the two key prerequisite materials for creating a Databricks workspace in a VPC that you manage? Answer: A VPC and a credential (principal) with the proper permissions.
Service accounts are managed under the workspace, so we don't need to create them right at the very beginning.
Connecting to Google Managed Services
Question: Which two tasks are part of the general pattern involved in connecting a Databricks workspace to Google managed services? (Meaning, we need to grant permission for the Databricks engine to access resources within your Google account). Answer: Enable the proper APIs and attach a privileged service account.
"Privileged" means having the correct permissions. We want to grant the Databricks engine permission to access and use our Google resources to deploy on the cloud server.
Why not other options?
Defining custom roles and enabling account console access: This is a data engineering job, but it is not the first thing we need to create.
Creating a storage credential and enabling APIs: This is done at the application level later; we don't need to create it right when we start the project.
Attaching a service account and creating a storage credential: Providing storage access to the Databricks engine is done at the workspace level, not as the very first step.
Encryption & Security
Question: Which elements can use encryption key configuration via Customer-Managed Keys? When we deploy sensitive data to a cloud service, encryption is a must to protect our data. Answer: The root storage bucket, the system storage bucket, and custom data volumes.
Root Storage Bucket: Protects personal data and metadata about what the data engineers can access (protects traffic in transit/at rest).
System Storage Bucket: Encrypts the data stored within the Databricks engine (data at rest).
Custom Data Volume: Refers to the hardware/memory on the instance server. We need to encrypt the entire server volume as well.
Why not other options?
External Storage Buckets: Databricks cannot be configured to encrypt external data outside of its own engine.
Metastore Bucket: The metastore holds background/metadata information attached to datasets for analytics. It should not store highly sensitive personal information (PII).
Question: Who is the access point for granting permission to the external storage bucket? Answer: A service account generated by the metastore creation. You can use this generated service account as an outgoing permission attachment.
Cost Optimization Strategy
Question: Which tools or strategies help reduce costs in a GCP Databricks platform? Answer: Custom tags, correct instance types, and auto-scaling.
Auto-scaling: If we only have one server to handle traffic, and a heavy workload comes in, the system will automatically scale out and create more servers to handle the ad-hoc workload.
Correct Instance Type: This means provisioning the appropriately sized resources for the job. We don't want to use massive resources for small, simple jobs.
Custom Tagging: If we have a server dedicated to data analytics, we tag it as such. This helps us understand what kind of job the server is doing so we can tie the proper instance size to that specific workload.
Why not other options?
VPC Architecture: This is primarily for security and isolation (e.g., a private VPC for internal sensitive data, a public VPC for third-party users), not cost optimization.
Cluster Tagging & Slack Integration: These are great for infrastructure monitoring and health checks, but not the best direct mechanism for cost optimization.
Workspace & Storage Requirements
Question: What are the workspace bucket requirements? Answer: We need to create two buckets: one for the system and one for DBFS (Databricks File System).
You can think of a workspace bucket as the storage hardware for your company data. We save business data on DBFS, but the Databricks engine needs its own separate storage. Therefore, we must isolate the system bucket from the DBFS bucket.
Why not other options?
Buckets are not shared across workspaces. Each workspace is a standalone environment. For security, we cannot share internal data with a third-party workspace, so separating workspaces is a high priority.
Question: What is true of an external location capability? Answer: It provides the ability to control access to a portion of an external storage bucket.
This allows for fine-grained access control. For example, if you hire a third-party analytics team, they only get access to common information. But if you hire a security firm to audit your architecture, you might grant them access to sensitive data. Fine-grained (role-based) access control is the best practice when sharing data.
Identity, Access & VPC Networking
Question: Which mechanism enables a Databricks cluster to access Google-managed services? Answer: Service account. It acts as the connection point for Databricks to access Google Cloud services.
Why not other options?
OIDC: This is an authentication mechanism, not a service access control.
VPC Peering: This connects two isolated VPC infrastructures to communicate with each other, but it is not an access management mechanism.
API Settings: This controls which users/applications can hit your Databricks API endpoints.
Question: What authorization scheme is required to query a list of workspaces programmatically? Answer: OIDC. Databricks uses OIDC as its modern authorization mechanism. (Base authorization using just a username/password is not recommended).
Question: What does the term "Standalone VPC" refer to? Answer: A VPC that resides in the same project as the workspace resources.
Question: Which mechanism registers a VPC into your Databricks account? Answer: Network configuration. A VPC (Virtual Private Cloud) is an isolated infrastructure layer, so network configuration is the proper way to define it.
Question: Where is the regionality determined when setting up a workspace in your own VPC? Answer: The Subnet. Within a VPC (Level 2 infrastructure), we have subnets (Level 3). You can deploy one subnet in the US and another in the UK. If a network issue takes down the UK subnet, high availability ensures you can still access the dataset from the US subnet to keep the business running.
Query Federation & Metastore
Question: Which two statements are true regarding Query Federation? Answer: Data can remain in place and is accessed with a read-only connection.
If you have datasets on Google Cloud or AWS, you don't need to copy the entire dataset into Databricks. Databricks can directly connect and read the data in place. Since it is querying an external system, it only has read permissions and cannot modify or update the external data.
Question: What are the prerequisites for creating a GCP Data Metastore? Answer: A bucket. The metastore is saved at the bucket level.
Question: What privileges do you need to perform queries using Query Federation? Answer: Metastore Admin and Account Admin. Workspace admins manage the infrastructure, but Metastore and Account admins handle the application level, including granting user access to datasets.
Flashcard Review Session
What enables Databricks to access GCP? A service account attached to the cluster.
What is the prerequisite for connecting to Google managed services? Enable APIs and attach a privileged service account.
What is Identity used for? To manage users or groups through an IDP (Identity Provider). We can reuse our on-premise IDP for Databricks to avoid managing multiple identity servers.
Who grants privileges on a data object? Only the data object owner can grant permission to external users.
Where are storage credentials created? In the Workspace. We use the Data Explorer to track access.
Who gets permission when granting access to external storage? The Databricks service account.
What is an external location? A secure access control layer over part of a storage bucket.
What defines a region in GCP Databricks? The subnet.
What is a Standalone VPC? A VPC residing in the same project as the workspace.
How do you register a VPC in Databricks? Using network configuration.
What are the prerequisites for a workspace in a custom VPC? The VPC itself, and a principal with the proper permissions.
What does encryption key configuration do? It uses a Cloud KMS key to encrypt data. The key can be rotated (e.g., yearly) to protect data from leaks.
What resources can be encrypted? The root bucket, system bucket, and custom volumes.
What is Query Federation? Querying external system data without moving it (read-only access).
How can you reduce compute costs? Using instance tagging, proper instance types, and auto-scaling.
How do you optimize query performance? Liquid clustering (an upgraded mechanism for fine-tuning server workloads), Z-ordering (for data indexing), and caching (for read-only queries).
What does Unity Catalog provide? Centralized governance and permissions. It acts as an indexing tool. We index the data (like schemas and tables), use it for data quality, and then run SQL queries.
(Note: There were questions in the chat about continuing the session. The answer is no, this concludes the interactive session, but feel free to ask any remaining questions about data engineering, data science, or machine learning on Databricks.)
Business Intelligence Architecture
When we create infrastructure, we need to think about costs (money in, money out) and business growth.
Looking at the Databricks architecture diagram:
Data Sources: We have datasets on-premise, on Google Cloud, AWS, or Azure. This includes files, logs, and IoT sensors. For basic business operations, we usually focus on files (like Excel) and logs first.
Ingestion: We run batch jobs or auto-loading.
Processing: This moves into feature engineering.
Cataloging: We use the Unity Catalog to index the data.
AI/BI Tools: We can use GenAI (LLMs) for business intelligence. We can use natural language queries. For example, if we want a summary of user behavior in the UK, we can train an AI job to query the data from the catalog and the on-premise files. We can also integrate external models from platforms like Hugging Face to generate summaries and executions.
AI/BI, Application Logic, & Media Handling
This is an important feature when we use Databricks to do business intelligence.
For the business application side, if we have a system, we get a lot of datasets from the transactional daily operations of the DBMS (Database Management System). For these database records, we can use Autoloader and database streaming. For this kind of database system, we can still use Genius (Databricks AI/BI).
It's still the same workflow: we can use Hugging Face to create the logic on our system. This is similar to using Cursor; you can apply Genius AI alongside your low-code IDE tools to create the application logic and handle the development jobs.
What about media (images or video)? Doing image or video processing jobs on Databricks is very expensive. This is not the key feature of Databricks, because Databricks is mainly built for AI and BI data jobs.
For media, you can still use Databricks to do the analytics, but if your company has experts on Google Cloud or AWS, it is better to directly store the media files on those platforms. For example, AWS SageMaker is a great option for doing machine learning and data analytics jobs specifically on media.
Data Ingestion & BI Tools
As you can see, for files—for example, Excel files or data exported from an application—we can use Autoloader to store everything in the catalog. Then, we use Genius with an AI model to process everything, which works perfectly.
What about Looker? Looker is primarily used with BigQuery on Google Cloud, so it is not directly related to Databricks. If you have Google Ads data integrated with Looker, that's perfect.
However, when you use Databricks, you should use Databricks Genius BI. It allows you to directly get the outcomes and execution logs you need to fulfill your business goals, instead of doing a lot of ad-hoc jobs back-and-forth across your data team and application team. You avoid doing a ton of manual work just to get the final outcome.
Batch vs. Streaming & Databricks IQ
Batch and Streaming are the underlying mechanisms when you use Databricks for ingestion:
If you want real-time analytics, you use streaming.
However, if you only want to run weekly or monthly jobs, batch processing is the right choice.
Next is Databricks IQ. This is the built-in AI assistant. You can treat it as a smart assistant that has similar, very helpful AI features to assist your workflows.
Conclusion & Q&A
Okay, this is the end of the presentation.
Regarding Unity Catalog access control, that is covered in the fundamental Databricks course, so I won't cover it again here. If you want to learn more features about that, I highly recommend checking out the Databricks Fundamentals course to keep track of it.
I'm so happy you joined me today, and hopefully, I will see you at the next meeting. Let's wrap up here. If you have any questions, feel free to ask me next time, and I'll be happy to share more Databricks infrastructure and data engineering best practices with all of you.
Does anyone still have questions?
If not, I will stay here for a few more minutes. If there are no further questions, I will end this session. Hope to see you next time!