Scaling the Architecture
All scaling operations should be done during maintenance windows. Keep the following points in mind:
When changing GKE node VM types, the node pool will be gradually replaced, which can result in downtime. GKE does this by:
Creating new nodes with the desired machine type within the same node pool
Cordoning and draining old nodes (safely evicting pods)
Terminating old nodes once pods are successfully rescheduled
Repeating this process in batches until all nodes are updated
When changing pod resourcing, some PVCs will not get replaced. For example, if a Kafka node has a persistent claim of 1TB and the new size calls for 2TB, the 1TB PVC will not be replaced without manual intervention.