How to Protect Sensitive Data in Databricks Without Limiting Analytics

What you choose in the protect step decides how hard everything else becomes. If you do nothing, finding and changing personal data is hard, because copies can sit in many places. With pseudonymisation, it is easier, because you can delete the pseudonym identifier. Anonymisation is easiest, but it may reduce the data's value.

Some parts stay with your organisation. Databricks does not cover consent management, privacy by design, the process for subject access requests, or incident response. These are organisational processes. The platform supports them, but it cannot replace them.

Finding personal data can now be partly automated. In May 2026, Databricks made automated data classification in Unity Catalog generally available, together with governed tags and attribute-based access control. It scans new data continuously and tags sensitive columns with built-in classifiers that follow standards such as GDPR and HIPAA. Custom classifiers were still in beta at that time.

Pseudonymisation vs anonymisation: which protection fits your data?

The course separates two approaches:

1. Pseudonymisation protects data at the record level. It replaces an identifier with an artificial value, and authorised users can still reverse it. Under the GDPR, a pseudonym is still personal data for the organisation that can reverse it.
2. Anonymisation protects a whole dataset. It changes the data irreversibly, so no one can identify a person directly or indirectly.

Pseudonymisation methods

Hashing applies a hash function such as SHA to the identifier after adding a secret random string, called a salt. Without a salt, an attacker can hash all likely values and compare the results. Store the salt in a Databricks secret scope, and give read permission only to production jobs and authorised users. Hashes need more bytes than the original values, so the data becomes larger and some operations become less efficient. If a downstream step needs part of a value, for example, the domain of an email address, hash that part into a separate column before you hash the full value.

One practical detail: the Spark function hash() returns a 32-bit, non-cryptographic value, so it is not suitable for this purpose. Use sha2 instead:

UPDATE prod.silver.users
SET email = sha2(concat(email, secret('pii-scope', 'email-salt')), 256)
WHERE user_id = 2;

Tokenisation replaces each value with a token and stores the original value in a secure lookup table, called a token vault. Writing is slower because of the lookup, but reading is fast, and the exposed table is smaller because it only holds the tokens. Only authorised users should have access to the vault.

Anonymisation methods

Data suppression removes information. You exclude columns with PII from views, remove rows of demographic groups that are too small, and use dynamic access control to give conditional access to the full data. Small groups matter. If only one customer lives in a small city, a regional report identifies that customer.

Generalisation removes specificity. The course lists four forms:

• Categorical generalisation: a city becomes a country.
• Binning: a birth date becomes a ten-year age range.
• Truncating IP addresses: the last byte becomes 0, which gives a /24 range.
• Rounding: a salary of 1245.4 becomes 1200.

In real projects, we usually combine several of these techniques.

Handling a deletion request: why deleting a record is not enough

Meeting a deletion request takes two steps, and both are required:

1. Delete the data in every table. Start in the first (bronze) layer, then pass the deletion on to the later layers (silver and gold). After this step, the person no longer appears in any query result.
2. Remove the old files from storage. A delete does not erase the old files. They keep the values until a cleanup job removes them, so until then the data is still on disk, and the request is not fully met.

Deletion is the most visible part of a compliance process for a data engineer. The current Databricks guide for GDPR and CCPA recommends this order: keep the deletion requests in a control table, delete the data in the bronze layer first with a scheduled job, propagate the deletion to silver and gold, and then remove the old files from storage. The guide also states that complete deletion is the safest option, because removing the risk of re-identification often requires deleting the personal data itself. If your tables use a surrogate key such as user_id instead of a natural key such as an email address, you can delete the PII columns and keep the non-PII data. This follows the course advice to remove natural keys.

Step 1: Delete the person's data everywhere it was copied

Structured Streaming treats its source as append-only. When a Delta table that a stream reads receives an update or a delete, the stream fails. The course shows three ways to handle this:

1. Ignore the changes. The option skipChangeCommits skips every commit that updates or deletes data and reads only the inserts. Databricks recommends it for most workloads that do not use the change data feed. The older option ignoreDeletes handles only the deletion of whole partitions and is now called legacy, and ignoreChanges is deprecated. Note that the stream then ignores the change completely, so you need separate logic to propagate it.
2. Change Data Feed (CDF). CDF records row-level changes between table versions. It is not enabled by default, and it adds some storage overhead for the change metadata. A downstream pipeline can then process only the changed rows. Databricks describes streaming from the change feed as the more robust implementation.
3. AUTO CDC in Lakeflow Spark Declarative Pipelines. It implements the ordering and deduplication logic for change feeds. CDF is specific to Delta tables, and AUTO CDC is the pipeline feature that applies a change feed to a target table.

The tool has a new name. Databricks renamed the APPLY CHANGES feature to AUTO CDC (in Python, create_auto_cdc_flow). The syntax is the same, and APPLY CHANGES still works, but Databricks recommends the new name.

A bulk deletion from a control table and the CDF commands from the course look like this:

MERGE INTO prod.bronze.users AS t
USING prod.privacy.deletion_requests AS r
ON t.user_id = r.user_id
WHEN MATCHED THEN DELETE;

ALTER TABLE prod.bronze.users
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

SELECT user_id, _change_type, _commit_version, _commit_timestamp
FROM table_changes('prod.bronze.users', 2, 5)
WHERE _change_type = 'delete';

The course recommends a separate deletion pipeline. New and updated records continue through the normal ETL pipelines, and delete events are filtered from the change feed and handled by a pipeline built for the privacy requirements. You can also add a commit message to the write, for example through the userMetadata option, so that the table history shows why the change happened.

The two table types in a pipeline behave differently:

• Streaming tables support DML. You can DELETE old records for a retention rule, UPDATE a column to scrub PII, or INSERT new data. The documented order is: delete in the source table, delete in the streaming table, and then read the source with skipChangeCommits. A full refresh of the streaming table is an alternative. Still, it reprocesses everything, so data that the upstream source already removed (for example, a Kafka topic that ages out after seven days) is lost.
• Materialized views do not support DML. They store the result of the last refresh. You delete at the source and refresh the view. Databricks documents that a materialised view handles source deletions automatically, but a source deletion can force a full recomputation, which can be expensive for large tables.

Step 2: Make sure the deleted data is really gone from storage

A DELETE creates a new table version, but the old files still contain the values. Delta keeps table history for 30 days by default, and the data files stay in storage until VACUUM removes them after the 7-day retention threshold. VACUUM also removes the CDF files in the _change_data folder. Deleting at a partition boundary makes the process more efficient. The same rule applies to pseudonymisation with UPDATE: the original values stay in old files until VACUUM runs.

Cleanup is now mostly automatic. For Unity Catalog managed tables, Databricks recommends Predictive Optimization, which runs VACUUM automatically. For tables without it, Lakeflow pipelines run maintenance within 24 hours after an update. You don't need to run VACUUM manually for routine work, but the retention period still determines when personal data leaves storage. In Databricks Runtime 18.0 and above, the retention of deleted files is controlled with the table property deletedFileRetentionDuration.

For an urgent erasure, the course shows this sequence: turn off the retention check, preview with DRY RUN, and run VACUUM with RETAIN 0 HOURS. Databricks recommends a retention of at least 7 days, because a shorter period can delete files that a long-running job has not committed yet. Use the sequence only when no long-running query or stream reads the table, and test it on a non-production table first.

There are three more points to check in a real deletion process. First, if the table uses deletion vectors, a DELETE only marks rows as removed. You must run REORG TABLE ... APPLY (PURGE) before VACUUM, and this also applies to materialized views and streaming tables. For Delta tables, deletion vectors are enabled explicitly or through a workspace setting whose default depends on the region. Second, a cluster with disk cache enabled can still hold data from files that VACUUM deleted until the cluster restarts. Third, the GDPR and the CCPA apply to all copies of the data, so you must also delete the data in upstream sources such as Kafka topics and cloud storage folders.

DELETE FROM prod.silver.users WHERE user_id = 5;

REORG TABLE prod.silver.users APPLY (PURGE);  -- only for tables with deletion vectors

-- urgent erasure without the 7-day wait
SET spark.databricks.delta.retentionDurationCheck.enabled = false;

VACUUM prod.silver.users RETAIN 0 HOURS DRY RUN;
VACUUM prod.silver.users RETAIN 0 HOURS;

SET spark.databricks.delta.retentionDurationCheck.enabled = true;

Isolating sensitive data: 5 layers that limit the damage of a mistake

Access control answers the question "who may read this object?" Isolation answers different questions: where is the data stored, which environments process it, and which credentials exist for it? A good design uses several independent layers, so that a mistake in one layer does not expose the data.

Everything starts with the Unity Catalog hierarchy. A metastore contains catalogues, a catalogue contains schemas, and a schema contains tables, views, volumes, models, and functions. Some securable objects belong directly to the metastore: storage credentials, external locations, shares, recipients, providers, and connections.

Layer 1: the metastore gives regional isolation only

An administrator creates one metastore per region and assigns it to one or more workspaces in that region. By default, the physical storage of a metastore is separate from the storage of other metastores in the account. This is regional isolation. However, the course states clearly that a metastore is not intended as a unit of data isolation. Every workspace assigned to it shares the same governance objects, so the boundary is too coarse. Data isolation should begin one level lower.

Two details changed since the early versions of Unity Catalog. Metastore-level managed storage is no longer required, so a design can use only catalogue-level storage. The metastore admin role is also optional, and Databricks recommends assigning it only if you need it.

Layer 2: the catalogue is the primary unit of isolation

A catalogue is the highest level that users work with, so it is the best place to separate data. Catalogues usually correspond to an environment (development and production), an organisational unit, or a sensitivity class (customer data and non-customer data). There are five reasons why this level works well:

• Inherited permissions. A privilege that you grant on a catalogue applies to all schemas and tables below it. This makes permissions short to define and easy to review.
• Two levels of visibility. USE CATALOG and USE SCHEMA are prerequisites for reading anything inside. Grant them only to people who should see the data. The BROWSE privilege is different: it lets people see the metadata for discovery and access requests, without any access to the data.
• Group ownership. Databricks recommends that groups, not people, own production catalogues and schemas, and that jobs run as service principals. It also recommends that only service principals have direct MODIFY access to production tables.
• Separate storage. A catalogue can use the metastore's storage, but the preferred option is a separate managed storage location for each catalogue.
• Workspace binding. A catalogue can be bound to specific workspaces. If production data should be processed only in the production workspace, the binding prevents queries from any other workspace, even from a user who has the SELECT privilege. Use it when the work environment and the data have the same isolation requirements. You configure bindings in Catalog Explorer, through the API, or with Terraform.

Layer 3: physical separation of storage

Some organisations require that certain data types are stored in specific accounts or containers of the cloud tenant. Unity Catalog can set the managed storage location at three levels: the metastore, a catalogue, or a schema, and the data is stored at the lowest level that is defined. With a catalogue-level location, all managed tables and volumes of that catalogue are stored in their own container, which can have its own network rules, encryption keys, and access list.

The documentation adds three rules for these containers. Use a dedicated container. Do not use a container that people or services can access from outside Unity Catalog, because that bypasses access control and auditing. Do not reuse a container that is or was used for the DBFS root.

For existing data, the building blocks are storage credentials and external locations. An external location is a path in cloud storage combined with a storage credential that Unity Catalog uses to access it. You use external locations to register external tables and external volumes. You can also bind storage credentials and external locations to specific workspaces, which adds another isolation boundary.

There is one condition that is easy to forget. If people can access the same container directly with their own cloud permissions, they can bypass every Unity Catalog rule. Limit direct access to these containers, and let Unity Catalog mediate all access. This also creates a complete audit trail of the requests. The current documentation gives more advice on this topic:

• Prefer managed tables for all new tables. They receive the automatic optimisations of the platform. Use external tables mainly for migrations from the Hive metastore, for disaster recovery requirements, and when external systems must read the files.
• Give permission to create external locations only to a few administrators.
• Do not give general READ FILES or WRITE FILES permissions on external locations to end users. Use volumes for path-based access to files.
• Do not mount a storage account to DBFS if it is also an external location.

Volumes cover the non-tabular data: libraries, configuration files, checkpoints, and any structured, semi-structured, or unstructured files. They can be managed or external. Note that you cannot register the files of a volume as a table. To query them as a table, you load them into a table.

Layer 4: compute isolation

The course describes three options:

• Single-user clusters protect one user's data, for example during a proof of concept.
• Multi-user compute supports users with different privileges safely. Examples are clusters in shared access mode and SQL warehouses.
• Multiple workspaces separate groups that do not collaborate.

The cluster modes have new names. Single-user is now called dedicated access mode, and shared is now called standard access mode. Databricks recommends standard access mode for all workloads and dedicated access mode only when a required function is not supported. Serverless compute and SQL warehouses support Unity Catalog. Use compute policies so that users can create only Unity Catalog-enabled compute.

Compute also matters for fine-grained controls. Row filters and column masks work only on compute that supports them (SQL warehouses, standard access mode, and dedicated access mode on recent runtimes), so check the requirements before you rely on them.

How Unity Catalog enforces the boundaries at query time

The security model lecture follows a query through eight steps, which explains why this design is safe:

1. A user or a service principal sends a query to a cluster or SQL warehouse.
2. The compute asks Unity Catalog to check the namespace, the metadata, and the grants. Unity Catalog logs the request.
3. Unity Catalog assumes the cloud credential that governs each object in the query.
4. It returns a short-lived, scoped token and a signed URL to the compute.
5. The compute requests the data directly from cloud storage with that token.
6. Storage returns the data.
7. The compute applies row and column filtering as the last step.
8. The compute returns the result to the user.

The important point is that the compute never holds a permanent storage key. It receives a temporary credential for exactly the objects that the query needs. This is why the best practice is to "avoid direct access to object stores".

Layer 5: rows and columns

The last layer limits what a permitted user sees inside a table. The course presents two mechanisms. Dynamic views filter or transform the data, and users query the view without any access to the underlying table. Row filters and column masks attach a SQL function directly to the table, which means you do not maintain a separate view for every case. Here is an example with the current Unity Catalog syntax:

CREATE FUNCTION prod.governance.region_filter(region STRING)
RETURN is_account_group_member('global-analysts') OR region = 'EU';

ALTER TABLE prod.sales.orders
SET ROW FILTER prod.governance.region_filter ON (region);

CREATE FUNCTION prod.governance.email_mask(email STRING)
RETURN CASE
  WHEN is_account_group_member('pii-readers') THEN email
  ELSE regexp_replace(email, '^[^@]+', '*****')
END;

ALTER TABLE prod.crm.customers
ALTER COLUMN email SET MASK prod.governance.email_mask;

The example uses is_account_group_member() because it is the recommended function with Unity Catalog. The course slide uses is_member(), which checks workspace-level groups, and it writes the row filter column without parentheses, which is not the correct syntax.

Policy-based access control is now generally available. In May 2026, attribute-based access control (ABAC) policies for row filtering and column masking became generally available in Unity Catalog. A policy is defined once at the catalogue, schema, or table level, refers to governed tags, and applies to every matching object below that level. A new table that receives the right tag is protected immediately, and automated data classification can apply the tags. The masking and filtering functions from this section remain useful because policies use functions for the logic. For a platform with many tables, start with ABAC policies and keep direct row filters and column masks for special cases.

Choosing a governance model: who is in charge of your data?

Unity Catalog supports two models. In the centralised model, a governance team owns the metastore and can own any object and grant or revoke any permission. In the distributed model, each catalogue, or set of catalogues, is a data domain. The owner of a domain creates and governs the assets inside it, independently of other domains. Both work, but in both cases Databricks strongly recommends that a group is the metastore admin or the catalogue owner.

A sample design

A project with personal data can use this structure:

• pii_prod: a catalogue with its own storage account, bound only to the production workspace. Only a small engineering group has access, jobs run as service principals, and no analyst reads it directly.
• analytics_prod: a catalogue with anonymised or aggregated data and masked views, available to a wide group of analysts.
• dev: a catalogue bound to the development workspace, containing synthetic or anonymised data only.

CREATE CATALOG pii_prod
  MANAGED LOCATION 'abfss://pii@<storage-account>.dfs.core.windows.net/uc';

GRANT USE CATALOG, USE SCHEMA, SELECT
  ON CATALOG pii_prod
  TO `pii-engineers`;

GRANT USE CATALOG, USE SCHEMA, SELECT
  ON CATALOG analytics_prod
  TO `analysts`;

All schemas and tables inherit the grants at catalogue levels, and each group receives only what it needs.

Encryption and new best practices

Databricks encrypts traffic between users, the control plane, and the compute by default (TLS), and encrypts stored data at rest (AES-256). You can add customer-managed keys if you need key control. The course slide lists AWS components such as EBS volumes. On Azure, the equivalents are the storage and disk encryption services, so check the Azure documentation for details.

The course ends the lecture with four practices that work as a review checklist. Three of them are now even stronger recommendations than in the course:

1. Avoid direct access to object stores. Fewer keys mean fewer places where a key can leak. Unity Catalog mediates access, and only infrastructure administrators need access to the object store.
2. Do not create new data assets in the Hive metastore. It is deprecated; all workspaces should migrate to Unity Catalog.
3. Store machine learning models in Unity Catalog schemas. The workspace model registry is now a legacy feature, and MLflow 3 uses the Unity Catalog registry by default. The slide title says "leverage workspace-local model registries", but the slide text says the opposite, so the text is correct.
4. Avoid DBFS. Both the DBFS root and DBFS mounts are deprecated, and new accounts are created without them. Use volumes for checkpoints, libraries, and configuration files, and external locations for cloud storage.

Audit evidence: how to show who accessed sensitive data

Isolation is only useful if you can verify it. Unity Catalog exposes system tables in the system catalogue, and you query them with normal SQL. The course covers four areas: object metadata (system.information_schema), billing (system.billing.usage), audit logs (system.access.audit), and lineage (system.access.table_lineage and system.access.column_lineage). System tables have a free retention period of 365 days.

Check the status of the audit tables. In September 2026, the documentation still lists the audit log table as Public Preview, while the billing and lineage tables are not marked as preview. Ask your administrator whether the schemas are enabled in your account. The audit log also hides some SQL definitions in request_params unless you are an account admin or a member of the databricks_pii_access group.

Two examples are useful for PII tables:

-- Who has privileges on the table?

SELECT grantee, privilege_type
FROM system.information_schema.table_privileges
WHERE table_catalog = 'pii_prod'
  AND table_schema = 'crm'
  AND table_name = 'customers';


-- Who requested data access to the table in the last 7 days?

SELECT user_identity.email, count(*) AS requests
FROM system.access.audit
WHERE service_name = 'unityCatalog'
  AND action_name = 'generateTemporaryTableCredential'
  AND request_params.table_full_name = 'pii_prod.crm.customers'
  AND event_date >= current_date() - INTERVAL 7 DAYS
GROUP BY 1
ORDER BY 2 DESC;

The query filters on event_date because the documentation recommends it for query performance, and action names can change, so check the audit log reference before you build a report on them.

Lineage is captured automatically for tables, columns, dashboards, jobs, notebooks, files, and models, and it answers two questions that matter for privacy: where did this data come from, and which downstream assets does a change affect? The course also recommends comments and tags on ingest. Today, governed tags are the better choice; for example, pii = ssn on a column, so that classification, search, and ABAC policies all use the same vocabulary.

Handling personal data: 4 principles to follow

The summary lecture gives ten rules. They can be grouped into four ideas:

1. Reduce what you keep. No PII is better than PII. Anonymisation is better than pseudonymisation, and pseudonymisation is better than clear text. The Databricks GDPR guide goes one step further: complete deletion is the safest option.
2. Question your own protection. Apply the "three facts rule": three pieces of information, such as postal code, birth date, and gender, can be enough to single out a person. Consider how datasets could be combined. The Netflix Prize studies in the resource list show how a dataset without names was matched with public movie ratings to identify subscribers.
3. Involve people. Train the data teams on the applicable privacy laws. Not all PII has the same risk, so run regular privacy impact assessments.
4. Isolate. Isolate the environments that process PII, and isolate the environment that protects it.

The course closes with a customer example from SEEK, an online employment marketplace in Australia that processes millions of resumes. They built a PII detection framework with Hugging Face transformers for structured and unstructured data, added optical character recognition for documents such as passports, tracked model versions with MLflow, and used Unity Catalog for lineage and access control. It is a good example of automated governance instead of manual reviews. Unity Catalog now also offers built-in automated classification for a similar purpose.

Key takeaways

In one sentence: protect personal data by design, and prove it with evidence. Reduce the number of copies, choose the strongest technique that the use case allows, make deletion complete and physical, isolate data at the catalogue, storage, and compute levels, and audit the result with system tables.

Checking the course against the current documentation shows a second lesson. The principles in the course did not change, but the tools around them changed quickly: AUTO CDC, standard access mode, ABAC policies, automated classification, and the retirement of DBFS. A security design should therefore include a regular review of the platform documentation.

When designing a new pipeline, ask the isolation questions before writing the first transformation: which catalogue will hold this data, in which storage account, from which workspaces can it be queried, and which group owns it? These decisions are cheap at the beginning and expensive after production data exists.