Add more S3 regions

[BenTheElder]

South America is the obvious missing one, but we should also consider adding others.

xref: kubernetes/k8s.io#4739 (comment) and discussion in https://kubernetes.slack.com/archives/CCK68P2Q2/p1678504635523609

/sig k8s-infra
/priority important-soon

When we add a bucket it will have to be added to the terraform config in k8s.io including the s3 to s3 replication, then once the bucket is reasonably well populated we'll want to add it to both https://github.com/kubernetes/registry.k8s.io/blob/8408d0501a88b3d2531ff54b14eeb0e3c900a4f3/cmd/archeio/app/buckets.go (AWS clients) and https://github.com/kubernetes/k8s.io/blob/5821271f875cc970b5701983e5695a025b5d952f/infra/gcp/terraform/k8s-infra-oci-proxy-prod/terraform.tfvars (non-GCP non-AWS clients)

This is much more relevant after #147

[ameukam]

/assign

/area infra/aws
/milestone v1.27

[k8s-ci-robot]

@ameukam: The label(s) area/infra/aws cannot be applied, because the repository doesn't have them.

Details

In response to this:

/assign

/area infra/aws
/milestone v1.27

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

We should ideally have:

1. All regions with notable client usage within AWS, for the AWS IP / region => bucket mapping
2. All regions that map closely to our Cloud Run / AR GCP regions, for external clients with the Cloud Run instance => bucket mapping

We started with a good approximation of 1) in #39 / #72, but it might be worth revisiting.

For 2) we have an okay approximation reusing the same set from 1), but we know there's at least the South America gap and should definitely re-evaluate and select the remaining regions that would most closely match now that we've moved to sending non-AWS-non-GCP traffic to S3 generally.

[ameukam]

List of remaining regions we can add:

• ap-east-1
• ap-northeast-2
• ap-southeast-1
• ap-southeast-3
• ap-southeast-4
• eu-central-1
• eu-central-2
• ca-central-1
• sa-east-1

[BenTheElder]

There's an inflection point where adding a region is wasteful.

If we are matching the IP to region with 1) for users within AWS and it only represents a very small amount of traffic within AWS then it's probably not worth it for 1), more regions than Cloud Run is not useful for 2) since we lean on the GCLB routing to determine approximate region for external traffic.

[BenTheElder]

OK so we looked at the costs today and storage costs are likely going to be smaller than traffic in basically any AWS region with actual traffic.

So just adding them for 1) is going to be generally worthwhile.

xref: https://kubernetes.slack.com/archives/CCK68P2Q2/p1678915249434949

[BenTheElder]

ref: #181 for hold a bit on production changes