What We Realized from Constructing GovSlack
10 min read
Slack launched GovSlack in July 2022. With GovSlack, authorities businesses, and people they work with, can allow their groups to seamlessly collaborate of their digital headquarters, whereas holding safety and compliance on the forefront. Utilizing GovSlack consists of the next advantages:
- Helps key authorities safety requirements, corresponding to FedRAMP Excessive, DoD IL4, and ITAR
- Runs in AWS GovCloud knowledge facilities
- Permits exterior collaboration with different GovSlack-using organizations by means of Slack Join
- Gives entry to your individual set of encryption keys for superior auditing and logging controls
- Permits permission and entry controls at scale by means of Slack’s enterprise-grade admin dashboard
- Features a listing of curated functions (together with DLP and eDiscovery apps) that may combine with Slack
- Maintained and supported by US personnel
Earlier than the large launch, the Cloud Foundations staff spent virtually two quarters establishing the infrastructure wanted to run GovSlack.
GovSlack is the very first service Slack launched on AWS Gov infrastructure. Due to this fact we needed to spend a major period of time studying the variations between commonplace and Gov AWS and making modifications to our tooling and the platform to have the ability to run on Gov AWS.
On this weblog submit, we’re going to have a look at how we constructed the AWS infrastructure wanted for GovSlack and challenges we confronted. In case you’re desirous about constructing a brand new service on AWS GovCloud, this submit is for you.
How are GovCloud accounts associated to industrial accounts?
Not way back, Slack began transferring from a single AWS account to little one accounts. As a part of this challenge, we additionally made vital modifications to our world community infrastructure. You may learn extra about this within the weblog posts Constructing the Subsequent Evolution of Cloud Networks at Slack and Constructing the Subsequent Evolution of Cloud Networks at Slack – A Retrospective. We had been capable of make the most of most of our learnings into constructing the GovSlack community infrastructure.
To begin with, AWS Gov accounts don’t have any billing functionality. The sources within the Gov accounts will propagate their billing right into a linked shell industrial AWS account. While you request a Gov AWS account, a linked shell industrial AWS account is routinely created. Due to this fact the very first thing we needed to do was to request a Gov root AWS account utilizing our root payer industrial account. This was a prolonged course of, however not as a result of it was a technically tough factor to do—it was so simple as clicking a button on our root industrial AWS account. Nonetheless including the Gov Accounts to our current agreements with AWS did take just a few weeks. As soon as we had our Gov root account, we had been capable of request extra GovCloud accounts for our service groups. It’s value mentioning that GovCloud little one accounts nonetheless must be requested utilizing the industrial AWS API utilizing the create-gov-cloud-account name.
When a brand new GovCloud little one account is created, you may assume the OrganizationAccountAccessRole within the little one account by way of the GovCloud root account’s OrganizationAccountAccessRole (this function title could differ when you override the title utilizing –role-name flag).
Let’s have a look at what are these hyperlinks seem like in a diagram:
As we are able to see above, all our GovCloud sources prices are propagated to our root industrial AWS account.
How did we create GovCloud accounts?
As we mentioned above, we use the AWS organizations API and the create-gov-cloud-account name to request a brand new GovCloud little one account. This course of creates two new accounts: the GovCloud account and the linked industrial AWS account. We use a pipeline on the industrial facet for this portion of the method. Then the linked industrial AWS account is moved to a extremely restricted OU, so it’s blocked from creating any AWS sources in it.
We use a Jenkins pipeline within the AWS Gov partition to configure the GovCloud little one account. We are able to assume the OrganizationAccountAccessRole of the brand new little one account from the GovCloud root account as quickly as it’s created. Nonetheless our Gov Jenkins providers are situated in a devoted little one account. Due to this fact there’s a step on this pipeline that can replace the kid account’s OrganizationAccountAccessRole’s belief coverage, so it may be assumed by the Jenkins employees. This step should be accomplished first earlier than we are able to transfer on to different steps of the kid account configuration course of.
How can we separate GovDev and GovProd?
As talked about beforehand, one of many core compliance necessities for a GovCloud atmosphere was that solely US individuals could be licensed to the manufacturing atmosphere. With this requirement in thoughts we made the choice to face up two Gov environments, one being the manufacturing Gov atmosphere, identified internally as “GovProd”, and a second atmosphere, often called “GovDev”. The GovDev atmosphere will be accessed by anybody and take a look at their providers earlier than being deployed to GovProd by US personnel.
To make sure we have now full isolation between these environments, we have now approached the construct out utilizing a full shared-nothing paradigm, which permits the environments to function in fully totally different AWS organizations. The layer 3 networking mesh we use (Nebula) is totally disconnected, that means the networks are fully segregated from each other.
To archive this, we created two AWS organizations in GovCloud, and underneath every of those organizations, an equivalent set of kid accounts to launch our providers within the Dev and Prod environments.
Is that this actually remoted?
When a brand new little one account is created, we have to use the Gov root account for assuming the OrganizationAccountAccessRole’s into it for the primary portion of the provisioning as we mentioned right here. Since solely US personnel can entry the Gov prod accounts, solely US personnel are capable of entry the Gov root account, as this account has entry to imagine the OrganizationAccountAccessRole within the little one accounts. Due to this fact the preliminary provisioning of dev accounts additionally must run on Gov prod Jenkins, and US personnel are required to be engaged to kick off the preliminary a part of GovDev accounts creation.
Different challenges
GovProd additionally lacks some AWS providers, corresponding to CloudFront and public zones in Route53. Moreover, after we are utilizing the AWS CLI in GovCloud, we should cross within the –area flag or set the AWS_DEFAULT_REGION atmosphere variable with a Gov area because the AWS CLI at all times defaults to a industrial area for API calls.
Route53 and ACM
A few of our Gov providers use AWS ACM for the load balancer SSL certifications. We keep away from utilizing e-mail certificates validation as this doesn’t enable us to auto-renew expiring certificates. ACM DNS helps auto-renewal however requires public DNS information to take action. Due to this fact, we use the identical devoted industrial DNS account for validating our ACM certificates as properly. Entry to this industrial DNS account is restricted to US personnel.
Route53
AWS GovCloud doesn’t assist public Route53 zones. Nonetheless non-public zones are allowed. We created a GovDev and Gov Prod DNS account for internet hosting non-public Route53 zones. The Cloud Foundations staff creates VPCs in a set of accounts managed by us, then we use AWS Transit gateways to attach totally different areas collectively and construct a worldwide community mesh. Lastly these VPCs are shared into little one accounts to summary the complexity of establishing networks from utility groups. You may learn extra about how we do that in our different two weblog posts Constructing the Subsequent Evolution of Cloud Networks at Slack and Constructing the Subsequent Evolution of Cloud Networks at Slack – A Retrospective
The non-public Route53 zones we create are connected to the shared VPCs, in order quickly as a document is added to those zones, it may be resolved inside our VPCs.
Nonetheless since GovCloud doesn’t assist public DNS, we have to create these information on the industrial facet. Due to this fact, we created a devoted industrial AWS account for internet hosting public GovSlack DNS information. Entry to this industrial DNS account is restricted to US personnel.
How can we switch artefacts between industrial and GovCloud?
AWS doesn’t assist assuming roles between AWS commonplace and AWS GovCloud partitions. Due to this fact we created a mechanism to compliantly cross objects to GovCloud.
This mechanism ensures the objects are pulled into AWS GovCloud partition from the usual partition utilizing AWS IAM credentials. Credentials to entry the usual partition for pulling these objects are saved securely on the AWS GovCloud partition.
Terraform modules
We use Terraform modules for constructing our infrastructure as a group of interdependent sources corresponding to VPCs, Web Gateways, Transit Gateways, and route tables. We needed to make use of the identical modules for constructing our Gov infrastructure so we are able to preserve these patterns constant between AWS Gov and commonplace partitions. One key distinction between the industrial and Gov AWS sources are the sources ARNs. Industrial ARNs begin with arn:aws versus Gov ARNs begin with arn:aws-us-gov.
Due to this fact we needed to construct a quite simple Terraform module referred to as aws_partition. Utilizing outputs of this module, we are able to programmatically construct ARNs and uncover which AWS partition we’re in.
Let’s have a look at the aws_partition module,
knowledge "aws_caller_identity" "present"
knowledge "aws_arn" "arn_details"
arn = knowledge.aws_caller_identity.present.arn
output "partition"
worth = knowledge.aws_arn.arn_details.partition
output "is_govcloud"
worth = change(knowledge.aws_arn.arn_details.partition, "gov", "") != knowledge.aws_arn.arn_details.partition ? true : false
Now let’s have a look at a instance utilization,
module "aws_partition"
supply = "../modules/aws/aws_partition"
knowledge "aws_iam_policy_document" "instance"
assertion
impact = "Permit"
actions = [
"s3:GetObject",
]
sources = [
"arn:$module.aws_partition.partition:iam::*:role/some-role",
]
useful resource "aws_config_config_rule" "instance"
depend = module.aws_partition.is_govcloud ? 1 : 0
title = "example-rule"
supply
proprietor = "AWS"
source_identifier = "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
VPC endpoints
During the last three years Slack has been working very exhausting to utilize AWS’ VPC endpoints for accessing native AWS providers in our industrial atmosphere. They cut back the latency and improve the resiliency of our methods, whereas additionally lowering our networking prices.
With all these benefits, it’s very straightforward to imagine that it’s a easy transfer, however one obtrusive concern that we have now present in each the industrial and GovCloud transfer to VPC endpoints is that AWS doesn’t at all times assist all providers in all AZs. Very often we have now discovered that we have to assist the flexibility for methods to entry AWS providers each with and with out VPC endpoints, which at occasions can create summary edge instances that may be exhausting to account for.
Whereas AWS is continually releasing these VPC endpoints at a AZ stage, we nonetheless haven’t reached 100% of providers enabled for 100% of the areas/AZs we run our service in.
AWS-SSO
Whereas we had been constructing out the Gov atmosphere, we began through the use of IAM customers to bootstrap the Gov atmosphere, however this was solely ever going to be a short-term resolution. AWS just lately launched the AWS-SSO resolution into their industrial atmosphere and much more just lately of their Gov atmosphere. As this was an entire greenfield buildout it was an excellent alternative to experiment with new applied sciences and enhance our current implementation.
In contrast to AWS’ commonplace IAM roles, AWS-SSO permission units are an org-wide world (throughout your complete org, versus an account) useful resource, and this modifications how we construct and deploy them.
Since deploying AWS-SSO within the GovCloud atmosphere, we have now taken the learnings and back-ported it into our industrial atmosphere. Whereas we already had an current SSO system in place for entry to everything of our industrial AWS atmosphere, utilizing AWS-SSO has made this course of rather a lot smoother and simpler.
So what have we realized?
Rebuilding our complete community infrastructure gave us the flexibility to check our tooling, processes, and Terraform modules, and gave us the chance to make enhancements. We had been capable of clear up a large number of hardcoded values and alter issues to be extra reusable. We had been additionally capable of take a step again and have a deep dive into our processes, instruments, AWS footprint and achieve a better understanding of our platform as this entire course of gave us a chance to rebuild Slack from scratch.
