OpenShift (OCP)

An SCC is an OCP admission policy that controls what Linux-level privileges a pod can request at runtime: whether it can run as root, which UIDs are allowed, whether it can use host networking/PID/IPC, which volume types it can mount, and which Linux capabilities are permitted. Every pod is evaluated against the SCCs that its service account has access to; the admission controller assigns the most restrictive applicable SCC. K8s PodSecurityAdmission (PSA) enforces one of three profiles (privileged, baseline, restricted) at the namespace level with no per-pod granularity. SCCs are per-service-account and per-pod, significantly more granular. OCP also ships predefined SCCs (restricted-v2, anyuid, privileged, etc.) rather than requiring you to define the policy from scratch.

A Route is OCP's native HTTP/HTTPS exposure primitive, handled by the HAProxy-based Ingress Controller (Router). Key differences from K8s Ingress: Routes support three TLS termination modes — edge (TLS ends at the router, plain HTTP to the pod), passthrough (TLS reaches the pod; the router cannot inspect or terminate), and re-encrypt (router terminates TLS and opens a new TLS connection to the pod using a separate cert). Routes also support weighted routing between services for A/B testing, custom router annotations for fine-grained HAProxy behavior, and route sharding for multi-router setups. K8s Ingress is the portable standard and works with any Ingress controller. In OCP, both objects are supported and Routes can be created automatically from Ingress objects. Prefer Routes for OCP-specific features (passthrough TLS, weighted routing); prefer Ingress for portability across clusters.

An ImageStream is a layer of indirection over container image references. Instead of a Deployment pointing directly to registry.example.com/myapp:3.1.0, it points to an ImageStream tag (myapp:production). The stream tracks what digest that tag currently resolves to. When the underlying image updates — either a new push to the registry or a promotion via oc tag — OCP can automatically trigger a new rollout on any Deployment or DeploymentConfig watching that tag. ImageStreams also enable image promotion across environments: oc tag myapp:staging myapp:production points the production tag at the staging image digest without any re-build or re-push. They abstract the registry URL from the workload definition, making environment-specific registry differences transparent.

DeploymentConfig (DC) is an OCP-specific resource that predates the K8s Deployment. It offered features K8s lacked at the time: image change triggers (auto-rollout on ImageStream update), custom lifecycle hooks (pre/mid/post deployment), and recreate/rolling strategies similar to K8s. K8s Deployment has since caught up on most functionality. DCs were deprecated in OCP 4.14 and will eventually be removed. Use K8s Deployment for all new workloads. Migrate existing DCs using the migration guide in the OCP docs — most are straightforward, replacing DC-specific fields with Deployment equivalents. ImageStream triggers on Deployments are possible via annotations, though as noted they should be used deliberately.

OCP RBAC has two layers. API-level RBAC (from K8s): Roles and ClusterRoles define sets of verbs on resources; RoleBindings assign them to users or service accounts within a project; ClusterRoleBindings apply cluster-wide. Built-in cluster roles: admin (full project control), edit (deploy and manage apps, no RBAC changes), view (read-only). Pod-level SCCs are a parallel gate: even if a user has edit on a namespace, the pod's service account must have access to an SCC that allows the pod's security requirements. The two systems don't interact directly — RBAC controls what you can do with the K8s API; SCCs control what the pod can do at the OS level.

The pod's service account does not have access to any SCC that permits what the pod is requesting. Diagnose: oc describe pod <name> to see the specific SCC validation failure; oc get events for the admission error message. Understand what the pod needs — check securityContext in the pod spec for runAsUser, runAsGroup, privileged, capabilities. Then check which SCCs the pod's service account can use: oc adm policy who-can use scc anyuid (or the SCC you think is needed). Fix: determine the minimum SCC required. If the image must run as root, grant anyuid to the service account: oc adm policy add-scc-to-serviceaccount anyuid -z <sa> -n <ns>. Prefer a dedicated service account per workload rather than using the default SA.

S2I is a build strategy where OCP injects application source code into a prebuilt builder image that knows how to compile and run that language. The developer provides source code; the platform handles the Dockerfile-equivalent logic. A Java S2I builder, for example, runs Maven/Gradle, packages the artifact, and produces a final application image — without the developer writing a Dockerfile. Advantages over Dockerfile: builder images are maintained by the platform team (one place to update base image, security patches, JDK version); developers can't introduce insecure Dockerfile patterns; the build is reproducible and auditable. Disadvantage: less control — if your build doesn't fit the S2I convention, you fight the tooling rather than just writing a Dockerfile.

ResourceQuota sets hard limits on aggregate consumption in a namespace: total CPU requests, memory requests, number of pods, PVCs, etc. If a pod would cause the namespace to exceed a quota, it's rejected. LimitRange sets per-container defaults and maximums: if a container doesn't specify requests/limits, LimitRange injects the defaults; if a container requests more than the maximum, it's rejected. Without LimitRange defaults, pods without resource requests are BestEffort QoS — first evicted under node pressure. ResourceQuota on CPU/memory requires every pod to declare requests; without LimitRange defaults, pods that omit requests are rejected by the quota admission. The combination is required: LimitRange provides defaults so developers don't have to specify every field; ResourceQuota enforces the namespace ceiling.

A MachineConfig is a declarative description of what the RHCOS node's OS state should be: kernel arguments (e.g., enabling huge pages, disabling swap), systemd unit files (e.g., a custom service), files on disk (e.g., /etc/sysctl.d/ tuning, custom CA certs), and SSH authorized keys. The MCO renders MachineConfigs into an Ignition config and applies it by draining the node and rebooting it. Common use cases: adding corporate CA certificates to all nodes, kernel tuning for database or high-performance workloads, enabling specific kernel modules, deploying a custom systemd service for monitoring agents that must run on the host.

OCP inspects the repository to detect the language/framework (presence of pom.xml → Java, package.json → Node.js, etc.). Based on detection, it selects an S2I builder ImageStream from the cluster's catalog. It creates: a BuildConfig (S2I build with the source repo), an ImageStream for the output image, a Deployment (or DC in older versions) pointing to the ImageStream, and a Service exposing the pod. A first build is triggered immediately. The Route must be created separately with oc expose service.

OCP's rolling deployment (K8s RollingUpdate strategy) creates new pods with the updated image, waits for them to pass readiness checks, then terminates old pods. Controlled by maxUnavailable (how many old pods can be down at once) and maxSurge (how many extra pods can exist during rollout). Zero-downtime requires: a properly configured readinessProbe (without it, new pods receive traffic before they're ready), preStop hook with a sleep if the app needs time to drain connections, and terminationGracePeriodSeconds long enough for in-flight requests to complete.

OCP uses K8s Secrets natively: base64-encoded key-value pairs stored in etcd, projected into pods as environment variables or volume mounts. OCP adds: the integrated Secret Store (via External Secrets Operator for Vault/AWS Secrets Manager integration), and etcd encryption at rest (enabled in OCP 4.x by default for Secrets). The default approach risks: Secrets stored in etcd are only as secure as your etcd backup security; base64 is not encryption — anyone with oc get secret access reads the value; Secrets in env vars are exposed in oc describe pod output and container process lists. Prefer volume mounts over env vars for sensitive values.

Structured approach: (1) oc logs <pod> --previous — get logs from the last crash; most root causes are in the application logs. (2) oc describe pod <pod> — check Events for OOMKilled, Liveness probe failures, volume mount errors, image pull failures. (3) If the container starts then immediately exits, use oc debug pod/<pod> to open a shell in a copy of the pod with the entrypoint overridden — inspect the filesystem, env vars, and volume mounts. (4) Check resource limits — OOMKilled means the container hit its memory limit; increase the limit or fix the memory leak.

etcd is the distributed key-value store that holds all of OCP's cluster state: every Deployment, Pod, Secret, ConfigMap, CRD instance, RBAC policy, and Operator configuration. Without etcd, the cluster cannot function — the API server is a stateless façade over etcd. etcd runs as a three-node raft cluster on the master nodes. Losing two masters (or etcd quorum) makes the cluster read-only. etcd backup (etcdctl snapshot save) captures a point-in-time snapshot of the entire cluster state. It's the primary DR mechanism: restoring from backup to a rebuilt control plane brings back all workloads, config, and RBAC. Without a backup, a failed cluster means rebuilding everything from scratch.

Cluster-level design: a namespace naming convention (team-environment, e.g., payments-prod), a MachineConfigPool per node class if teams need node isolation, and NetworkPolicy defaults in the project template to deny cross-namespace traffic by default. Per-team onboarding: create projects (dev, staging, prod) via a GitOps-managed namespace provisioner (not manually); apply ResourceQuota and LimitRange from a template; bind the team's RBAC group to edit in dev/staging and a custom restricted-edit (no secret reads) in prod; provision a dedicated service account per workload type with the minimum required SCC pre-granted.

OpenShift GitOps ships ArgoCD as a cluster Operator. Install via OperatorHub; the operator manages ArgoCD instances. Define an Application CR pointing to a Git repo and a target namespace. ArgoCD continuously reconciles Git state → cluster state. For multi-environment pipelines: one Git repo with overlays per environment (Kustomize) or values per environment (Helm); promote by updating the image tag or Kustomize overlay in Git — the merge to the environment branch triggers ArgoCD sync. For secrets: use External Secrets Operator to pull from Vault; never store raw secrets in Git.

OCP 4.12+ defaults to OVN-Kubernetes as the CNI. OVN (Open Virtual Network) implements networking using kernel-level OVS (Open vSwitch) flows managed by a control plane. It supports NetworkPolicy natively, egress firewall, hybrid networking (Windows nodes), and better performance than the older SDN. Each node runs an OVN agent that programs OVS flows. OpenShift SDN (legacy, deprecated): a simpler model using VXLAN overlays with three modes (subnet, multitenant, networkpolicy). Less flexible but simpler to reason about. Multitenant mode provided implicit namespace isolation that NetworkPolicy requires explicit rules to replicate. Migrate from SDN to OVN-K during upgrades — SDN is removed in OCP 4.17.

Prerequisites: check the upgrade graph (oc adm upgrade), confirm the target version is in the supported path (no skipping minors), verify all cluster operators are healthy (oc get co), ensure etcd backup is current, and review the release notes for deprecated APIs. Process: oc adm upgrade --to=<version> triggers the CVO. The CVO upgrades the control plane first (masters one at a time via MachineConfigPool), then worker nodes (drain, upgrade RHCOS, reboot, uncordon). Worker upgrades are rate-limited by maxUnavailable in the worker MachineConfigPool — tune this for your workload tolerance.

Operators codify operational knowledge as code — they're appropriate when: the lifecycle of the application requires complex state machine logic (install, configure, upgrade, backup, restore are all distinct states with failure recovery); you need to react to runtime events and self-heal (e.g., a database operator that detects a failed replica and triggers recovery automatically); or you're building a reusable platform capability that other teams will consume via a simple CRD interface. Helm is right for: application deployment packaging where lifecycle is handled by external processes (CI/CD pipeline controls upgrades); third-party software with existing charts; environments without Operator development expertise. Plain manifests are right for: simple stateless applications with no operational complexity.

Two patterns. External Secrets Operator (ESO): deploy ESO from OperatorHub; define a SecretStore (Vault credentials, address) and ExternalSecret CRs that map Vault paths to K8s Secret keys. ESO reconciles on a schedule — the Secret is kept in sync with Vault, rotating automatically. Vault Agent Injector (Sidecar): Vault injects a sidecar that authenticates to Vault via the pod's service account (Vault K8s Auth) and writes secrets to a shared in-memory volume at /vault/secrets. The application reads files, not env vars — rotation is handled by the sidecar without pod restart.

Build once, promote by reference — never rebuild the same image for each environment. Pipeline: (1) CI builds the image on code merge, pushes to a dev registry with a commit-sha tag. (2) Dev namespace ImageStream points to the commit-sha tag; automated tests run. (3) On test pass, promote: oc tag myapp:commit-sha myapp:staging — the staging ImageStream now references the exact digest that passed tests. (4) After staging validation, oc tag myapp:staging myapp:production. Each environment's Deployment watches its environment-specific ImageStream tag; the image is the identical binary at each stage.

OCP ships a full Prometheus + Alertmanager + Grafana stack via the Cluster Monitoring Operator. It monitors cluster components out of the box (API server, etcd, nodes, network). User Workload Monitoring (UWM) is a separate Prometheus instance for application metrics — enable it via the cluster-monitoring-config ConfigMap. Key cluster health signals: cluster operator status (oc get co), etcd wal_fsync_duration_seconds (latency), node resource pressure (disk, memory, PID), API server request latency and error rate, and pod scheduling latency.

ACM (Red Hat Advanced Cluster Management for Kubernetes) is a hub-and-spoke multi-cluster management plane. A hub cluster runs ACM; spoke clusters are imported and managed via a pull-based agent. ACM provides: fleet-wide policy enforcement (OPA/Gatekeeper policies deployed cluster-wide), application deployment across clusters (Subscription model or GitOps integration), cluster lifecycle management (provision, upgrade, decommission), and observability aggregation across clusters. ACM justifies its complexity at three or more clusters — below that, per-cluster management is simpler. Above five clusters, manual per-cluster operations become unsustainable, and consistent policy enforcement without ACM requires significant process discipline.

OPA/Gatekeeper (the K8s Admission Controller backed by Open Policy Agent) is the most flexible: write Rego policies as ConstraintTemplates, instantiate them as Constraints. Example: deny any pod that requests privileged: true, require all Deployments to have resource requests. ACM integrates with Gatekeeper to distribute policies fleet-wide. Kyverno is an alternative with a simpler YAML-based policy language — better for teams without Rego expertise; supports mutation (automatically adding labels, resource defaults) as well as validation.

OCP supports stateful workloads via StatefulSets + PersistentVolumeClaims. Storage is provisioned via StorageClasses backed by CSI drivers (Trident for NetApp, EBS CSI for AWS, vSphere CSI, Ceph/OpenShift Data Foundation). Key considerations: choose a StorageClass with the right access mode (ReadWriteOnce for single-pod databases, ReadWriteMany for shared filesystems); set reclaimPolicy: Retain for production PVCs so a PVC delete doesn't destroy data; size PVs with growth headroom; use Pod Anti-Affinity to spread replicas across nodes/zones.

Start with a default-deny NetworkPolicy in every namespace: no ingress or egress allowed unless explicitly permitted. Then layer allow rules: pods within the same namespace can communicate freely (namespace selector); allow ingress from the OCP router (HAProxy) to any pod exposing a Route; allow egress to the cluster DNS (port 53 to the DNS service IP). Per-team: each team adds explicit ingress rules for cross-namespace traffic they need (e.g., monitoring namespace can scrape metrics on port 8080).

IPI on EC2 — you own everything: the installer provisions EC2 instances, VPC, ELBs, IAM users, Route 53 entries, and security groups. You operate the control plane (master nodes are yours, etcd backup is yours, upgrades are yours). Full flexibility: choose any instance type, customize the VPC layout, deploy into an existing VPC (existing VPC install), configure private clusters, air-gapped installs, or custom AMIs. Control plane costs are visible EC2 spend. Requires platform engineering bandwidth for cluster operations. ROSA — Red Hat operates the control plane (masters are on Red Hat's AWS account, not yours). You manage worker nodes and workloads. ROSA uses AWS STS for all IAM interactions — no long-lived credentials on the cluster. Upgrades are managed with a maintenance window policy; Red Hat responds to control plane incidents. ROSA integrates natively with AWS services: IAM roles for service accounts (IRSA), PrivateLink for private cluster access, CloudWatch log forwarding. Faster to provision (30–45 minutes vs. 90+ for IPI).

Single cluster breaks down when: you need blast radius isolation between business domains (a platform incident in one domain shouldn't affect others); regulatory requirements mandate data residency in separate regions; scale exceeds what a single cluster can support cleanly (~500 nodes is practical maximum before management overhead grows); or upgrade cadence differs between teams. Multi-cluster patterns: hub-spoke (ACM managing N spoke clusters), each spoke scoped to a team, environment, or region; tiered (one cluster per environment: dev, staging, prod); federated (multiple prod clusters behind a global load balancer for geo-distribution and DR).

Evaluate on five axes. Compliance: does the target regulatory framework (FedRAMP, HIPAA, SOC2, PCI) require specific certifications or on-premises data residency? OCP has the most certifications and runs on-prem; managed services are cloud-only. Operational model: what's the team's expertise and appetite for platform engineering? OCP provides more capability but requires more engineering depth. Cost at scale: OCP licensing is per-core and significant; EKS/GKE/AKS charge per cluster and per node. At large scale, total cost can favor OCP; at small scale, managed wins on simplicity. Ecosystem alignment: Red Hat ecosystem (RHEL, Ansible)? AWS ecosystem? This matters for tooling integration and support contracts. K8s version currency: OCP lags; managed services are current.

Platform engineering for OCP is a product discipline, not an ops function. The platform team's outputs are: a self-service onboarding flow (new team → namespace, quota, RBAC, CI/CD pipeline template in <30 minutes via automation); a curated operator catalog (vetted, supported operators from OperatorHub available to teams); a base image supply chain (RHEL-based builder images, patched on a schedule, promoted through a registry pipeline); a GitOps framework (ArgoCD, standard Helm/Kustomize conventions, secret management pattern); and an observability contract (UWM enabled, dashboards available per namespace, alerting SLAs defined).

Operators are OCP's primary extension mechanism — the platform is operator-driven, the ecosystem delivers software via operators, and teams are encouraged to write operators for their domain. The benefits: lifecycle management as code, self-healing, declarative APIs via CRDs, consistent operational patterns across the org. The costs: Operators add a controller process per application that must be operated itself (what happens when the operator crashes?); CRD API versions must be maintained (deprecation and migration); operator upgrades can break CRD contracts; and the skill set to write production-quality operators (controller-runtime, reconciliation design, status condition handling) is non-trivial.

OCP DR has two components: control plane recovery (etcd restore) and workload recovery (PV data + Deployment manifests). For the control plane: etcd snapshots on a schedule (hourly for prod), stored externally (S3/NFS), tested quarterly with a full restore drill on a non-production cluster. For workloads: GitOps-managed manifests mean Deployment state is in Git — cluster rebuild + ArgoCD sync recovers stateless workloads. Stateful data (PVs) requires a separate backup strategy: Velero for PV snapshots, or application-level backup (pg_dump for PostgreSQL). RPO drives backup frequency; RTO drives automation. A 1-hour RTO requires automated cluster provisioning (IPI or UPI scripted) + automated restore, not a manual runbook.

OCP is licensed per core (physical or virtual). Licensing cost grows with cluster size — right-sizing nodes matters. Optimization levers: use compute nodes with higher core counts (fewer license units per workload unit); autoscale worker nodes (Machine Autoscaler in OCP reduces idle capacity); bin-pack workloads by setting accurate resource requests to maximize pod density per node. For capacity planning: track namespace resource requests vs. actual usage per team (Prometheus kube_pod_container_resource_requests vs. container_cpu_usage_seconds_total); charge-back by namespace to make teams accountable for waste.

Conway's Law states systems mirror the communication structures of their builders. In OCP: cluster and namespace boundaries will reflect your org chart whether you design it that way or not. Teams that own a service end up owning a namespace. Clusters often align to org units (infrastructure cluster, payments cluster) because security and operational requirements differ by org boundary. The risk: if team boundaries are wrong (a monolith team that should be split, two teams that should merge), the namespace and cluster boundaries encode that organizational dysfunction and make it harder to change.