Cloud KMS for Data Engineering

Master Cloud KMS including key management, CMEK, CSEK, key rotation, and encryption patterns for data engineering.

16 min readIntermediate

Key Management Architecture

🛡️ GCP Security Architecture for Data Engineering

Interview Tip: GCP follows a shared responsibility model — Google secures the infrastructure, you secure your data. Enable encryption at rest (default), use CMEK for sensitive data, implement VPC Service Controls for data exfiltration prevention, and use Cloud DLP to classify and protect PII.

Implementation

from google.cloud import kms_v1

client = kms_v1.KeyManagementServiceClient()

# Create key ring
key_ring = client.create_key_ring(
    request={
        "parent": "projects/my-project/locations/us-central1",
        "key_ring_id": "data-engineering-ring",
    }
)

# Create crypto key with automatic rotation
key = client.create_crypto_key(
    request={
        "parent": key_ring.name,
        "crypto_key_id": "bigquery-encryption-key",
        "purpose": "ENCRYPT_DECRYPT",
        "version_template": {
            "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION",
            "protection_level": kms_v1.CryptoKeyVersion.ProtectionLevel.HSM
        }
    }
)

# Enable automatic rotation (90 days)
key.rotation_period = {"seconds": 7776000}
client.update_crypto_key(request={"crypto_key": key})

# Encrypt data
def encrypt_data(key_name, plaintext):
    """Encrypt data using Cloud KMS."""
    response = client.encrypt(
        request={
            "name": key_name,
            "plaintext": plaintext
        }
    )
    return response.ciphertext

# Decrypt data
def decrypt_data(key_name, ciphertext):
    """Decrypt data using Cloud KMS."""
    response = client.decrypt(
        request={
            "name": key_name,
            "ciphertext": ciphertext
        }
    )
    return response.plaintext

CMEK for BigQuery

from google.cloud import bigquery

client = bigquery.Client()

# Create dataset with CMEK
dataset = bigquery.Dataset("my-project.encrypted_dataset")
dataset.location = "us-central1"
dataset.encryption_configuration = bigquery.EncryptionConfiguration(
    kms_key_name="projects/my-project/locations/us-central1/keyRings/data-engineering-ring/cryptoKeys/bigquery-key"
)

dataset = client.create_dataset(dataset, exists_ok=True)
print(f"Created dataset with CMEK: {dataset.dataset_id}")

✨

Best Practice: Use CMEK for all sensitive data in BigQuery and GCS. Enable automatic key rotation (90 days). Use HSM for highest security. Implement key destruction policies for data retention. Monitor key usage via audit logs.

💬

Common Interview Questions

Q1: What is the difference between CMEK and CSEK?

Answer: CMEK (Customer-Managed Encryption Keys) are managed via Cloud KMS with automatic rotation and lifecycle management. CSEK (Customer-Supplied Encryption Keys) are provided per-request and never stored by Google. CMEK is easier to manage; CSEK provides more control.

Q2: When should you use HSM vs. Software keys?

Answer: HSM keys provide hardware-backed security for compliance requirements (HIPAA, FedRAMP). Software keys are sufficient for most use cases. HSM keys cost more but provide higher security guarantees.

Q3: How does key rotation work?

Answer: Cloud KMS automatically rotates keys at configured intervals (default: yearly). Old key versions remain available for decryption. New encryptions use the latest version. Manual rotation is also supported.

Q4: What happens when a key is destroyed?

Answer: All data encrypted with that key becomes permanently unrecoverable. Cloud KMS implements a waiting period (default: 24 hours) before destruction. Use key destruction for data retention compliance.

Q5: How do you audit key usage?

Answer: Cloud KMS logs all key operations to Cloud Audit Logs. Monitor encrypt/decrypt operations, key creation/destruction, and IAM changes. Export logs to BigQuery for analysis. Set up alerts for unusual key usage.

Cloud KMS: Key Management, CMEK & CSEK