Standardize Deployments

Create Workspaces and Workloads

Problem and Solution Overview

This recipe just states how to execute a solution. Read our Legacy Newsletter blog post: DevOps #10: Deploy Independently with Workspaces and Workloads to understand the specific problem we are solving and the solution approach.

This recipe helps you verify that your code uses its dependencies correctly, without running any integration tests.

  1. Initial Purposes and Workspaces.
  2. Initial Workspace Standard Capabilities.
  3. Assemble the Hexagons.
  4. Develop Standards Iteratively.
  5. Workspaces to Avoid.

Initial Purposes and Workspaces

There are 2 kinds of Workspaces:

  • Fixed-purpose. These Workspaces serve a single Purpose.
    • They are optimized for that Purpose and usually highly-scalable or otherwise specialized.
    • They usually host a single Workload and update by upgrading that Workload to new versions.
    • They implement the full standardized interface and connect to all parts of the organization.
    • Deployments are usually triggered automatically.
    • State is usually migrated with each update to retain all data.
  • Ad-hoc. These Workspaces serve many different Purposes over their lifetimes, and often several Purposes at once.
    • They are optimized for low-cost flexibility and usually don’t scale well.
    • They often host multiple Workloads at once, and upgrade by adding and deleting Workloads as needed.
    • They implement a subset of the standardized interface and connect to only some parts of the organization, often in a simplified way.
    • Deployments are usually triggered manually.
    • State is usually replaced on each update, either with an error trapped from production or a clean system.

A good initial configuration is:

  • Per employee: 1 Ad-hoc Workspace.
    • Developers will use this to chase down bugs or to try out new resources and code options.
    • Sales people will use this to run demos.
    • Support people will use these to reproduce customer issues and try potential workarounds.
    • Marketers and Product Owners will use this to explore new features, provide feedback, and assemble release communications.
  • Per product: 2 Fixed-purpose Workspaces.
    • CI: used to do a single smoke test at the end of CI builds.
    • Production.

Your goal at this phase is to name the starting set of Workspaces.

Steps:

  1. Gather the minimal set of required parties.
    • Usually this includes at least one representative from: management, development, operations, and accounting.
  2. Align on definitions for Application, Purpose, Workload, and Workspace.
  3. Align on types of workspaces. Ensure everyone agrees on examples for both Fixed-purpose and Ad-hoc needs.
  4. Propose the recommended initial config described above.
  5. Gather and resolve objections.
    • Some objections will require modifying the set of Workspaces.
    • Most objections will identify required Workspace Capabilities. Record theses so you can build them out in the next phase.
  6. Agree on a naming convention for all Workspaces.
    • Make sure every name is concrete and obvious.
    • When you can’t find a concrete, obvious, and systemic name, then your system is not yet clear enough. Add, modify, or change Workspace boundaries until naming is easy.

Initial Workspace Standard Capabilities

The workspace defines capabilities needed to support 4 properties:

  • Deployable: we can deploy a Workload and have it run.
  • Secure: the Workspace is a bounded security context. Nothing in it has permissions to affect anything outside it — including the deployment of resources.
  • Accountable: the Workspace is a bounded cost context. We can report on its costs independently, and the organization can make independent budgeting choices for the Workspace.
  • Monitorable: the Workload can be monitored to make sure it is available.

Each of these properties will become more complex over time. For example, “secure” has many more aspects than just being a bounded context! However, these are a minimal set of capabilities that usually match the results the enterprise is getting from its prior, ad-hoc approach. As such, they are a good starting point.

These properties require coordination among three different job roles, often in three separate departments:

  • Operations
    • For Deployable: define resource group, resource config standards, and pipeline execution (not definition).
    • For Secure: define service principals, connect them to all components used to support Deployable.
    • For Accountable: define tags needed to aggregate costs correctly and ensure they will be automatically applied when Development allocates resources in their pipeline definition.
    • For Monitorable: create dashboard and monitoring tool (availability only).
  • Accounting
    • For Accountable: define reporting needs and communication channel used to discuss budgets and purchases.
  • Development
    • For Deployable: define the pipeline steps.

Steps:

  1. Operations choose the tool that will be used to manage resources.
    • Good choices include Terraform and Pulumi.
    • Ensure the tool includes the ability to define and run tests.
    • Ensure you can manage resources in all cloud providers and in “on-prem” or private data centers.
    • Ensure Operations can define reusable modules that Development then uses to allocate a resource — so that Development doesn’t need to define all the details correctly every time.
  2. Operations start creating modules to allocate resources and to test results.
    • Teams will use modules in their resource and test definitions. They don’t create resources directly; they use module instead.
    • The Ops team can use modules to share resources or provide alternate implementations in different Workspaces. This allows them to optimize cost and performance to match needs, independently from development.
    • For example, Ops may define a “WebApp” module. A Dev team uses that to set up an App’s host and install code. The module then sets up all required resources to make that work. For example, in the ad-hoc Workspaces, it might use a shared App Service / VM in order to minimize costs. But on Production it defaults to a stronger plan, and on one specific Workspace it might configure multiple high-memory VMs and a load balancer.
  3. Operations create the initial Workspaces.
    • These should be built automatically based on PRs to main in some repository.
    • Start with as empty a Workspace as possible. You will have to define it, but keep that definition empty for this step.
    • Triggers should perform an update whenever any of the following change: modules defined in step 2, set of workspaces, or definition of a workspace.
  4. Operations define the initial Workspace configuration (standard resources and interfaces).
    • Update the script in the repository incrementally to add one capability at a time.
    • Ensure that your triggers from step 3 cause each definition to auto-deploy to existing Workspaces.
  5. Operations define the default new project.
    • Include source control, empty CI pipeline, and anything else you need — except for the CD pipeline.
    • Ensure that creating a new project and pushing to main will execute CI, resulting in test execution and a build artifact.
  6. Operations define the default CD pipeline.
    • Any new project gets this as a starting point.
    • It does nothing, but makes clear how Development can add capabilities.
    • Operations ensure that creating a new project automatically creates this pipeline and starts executing it.
    • It should automatically execute under the right conditions. For example, each successful CI build should automatically trigger execution to the CI Workspace, and manual execution should trigger execution to the Production Workspace.
  7. Development start building a pipeline definition.
    • This is checked in to the application’s source control, such that it is kept in sync with code and the CI pipeline definition.
  8. Operations add tags and reports to resource modules.
    • Modifying these modules should automatically trigger CD modifications on all Workspaces to update all allocated resources.
    • Add the tagging last to ensure that your automated triggers from step 3 do perform those updates automatically.
  9. Accounting create the accounting channels.
    • Create a specific channel for each separate budget.
    • Map those to the Workspaces, so that each Workspace simply has one channel and it always has the same name. Multiple Workspaces may share the same channel, if they share the same budget.
    • For example, inside each Workspace there may be a link to “accounting conversations” which goes to some Slack channel. Accounting sees several channels, each named according to its budget / purpose. All others simply see “the accounting conversation for this Workspace.”
    • If necessary, split budget and spending into two separate channels per budget. Usually this only adds complication, but sometimes it is needed.
  10. Accounting and Operations create accounting reports.
    • Use the tags to aggregate resource costs and define the necessary cost reports.

Assemble the Hexagons

The core operation is to run a Workload in a Workspace. We support that with operations including:

  • Deploy a Workload to a Workspace.
  • Create a new Workspace.
  • Modify a Workspace to meet new system-level needs.
  • Create a Workload by defining a Purpose and then assembling Applications.
  • Modify an Application and update associated Workloads.
  • Merge Workloads.
  • End-of-life a Workload.
  • Delete a Workspace.

Each of these operations should be fully automated, though each also has different triggers, gates, and owners.

For example, the “create a new Workspace” operation might be owned by a central DevOps team. It might be triggered by a pull request (PR) to main in their workspaces repo, and that PR might require sign-off (a gate) by the DevOps team to support it and an accounting authority to fund it. However, once that PR is approved and merges to main, the rest of Workspace creation is fully automated.

Steps:

  1. Identify one supporting operation at a time.
  2. Identify the following parts for the operation:
    • Owner: which team performs the core of the operation. These people see this operation as their work.
    • Gates: which people or circumstances need to be able to block or delay the operation.
    • Triggers: which events or actions trigger doing the operation.
    • Contributors: which teams, people, or systems provide information or content that the operation operates on.
  3. Define where the Owner will maintain the work product for the operation.
    • Ensure you can handle updates and changes but always know which is the current version.
    • Strongly consider source control. It is usually cheaper to train people to use SourceTree than it is to train people in all the manual processes needed to manage files correctly with shared folders.
    • Ensure the work product storage system is standardized enough that you can connect it to automation.
    • Keep all “source” products (changed by humans) separate from “built” products (changed by automation).
  4. Identify all interactions between the Owner and Contributors.
  5. Automate every interaction in step 3.
    • A policy or procedure is not enough. Those create too much overhead. Automate it.
    • Export only “built” products.
    • Use existing entry points wherever possible. For example, each Workspace can have a single internal website. Each operation can then link off from that website so that everyone only has to learn “go to the Workspace’s website for any operation.”
  6. Create a sensible default for each interaction.
    • An empty project needs to fully support the standard without the dev team taking any action. The team will incrementally add concerns during development.
    • Teams will worry about compile and test concerns immediately, but won’t think about things like supportability for months or even a year or two.
    • The Workspace and organization needs to support the team by not forcing them to consider every concern at once. Provide a sensible default so that the organization can interact correctly with the product even though the development team is ignoring the specific organizational need for now.
    • Teams will also use these on existing products. The team wants to only worry / think about exceptions. Sensible defaults allow the team to not worry about the many details that matter to the organization as a whole but where this team is following the norm. This reinforces consistency across projects and further reduces whole-organization costs.
  7. Automate the Gates incrementally. Use Checklist Automation.
  8. Automate each Trigger once its downstream Gates are in place.
    • Gates don’t need to be fully automated, but you do need at least the automatically-run checklist.

Develop Standards Iteratively

Workspaces are mechanism for standardization. In order to meet our goal of standardizing the right things, we need to develop Workspaces iteratively. We iterate on two dimensions at once:

  • Start with a small set of Workspaces and add and split them over time.
  • Start your Workspace boundary with a small set of capabilities and add them over time.

Iterating on the set of Workspaces is easy. These are all defined in a repository owned by some internal IT/Ops/DevOps team. The team simply modifies the config file, checks it in, and allows the automated deployment to update the workspaces.

Adding capabilities to Workspaces is covered in Assemble the Hexagons. The main challenge is to resist standardizing everything. Each standard added to the Workspace slightly increases the cost and complexity for every Dev team. Sensible defaults and shared entry points reduce friction but don’t completely eliminate it. Focus on the most important organizational needs and standardize those.

Be brutal. Typically half or more of the “organizational needs” are sacred cows or pet projects. The right answer is to cut them.

This means that some organizational needs will not meet the minimum bar to standardize into Workspaces. Those needs may be important for the organization, but less important than product development speed and efficiency. That’s OK, but it means that the organization needs to find a way to meet the need without involving product development or operations at all.

Usually the best solution is automation or outsourcing — the fact that these needs are less important than development efficiency means they likely are not a business differentiator. Make your company more Lean by freeing up all of your company’s personnel from this concern.

Workspaces to Avoid…and When to Add Them

  • Team-shared Workspaces (Ad-hoc or Fixed purpose).
    • These shared resources cause errors and delays. Use individual Ad-hoc Workspaces instead.
    • The usual objection is cost. However, you can always share low-level resources among Workspaces in a way that can’t conflict at the Workload level.
  • QA or Testing Workspace for a Product.
    • Use automated verification instead. Your automated verification will be sufficient if you have followed the recommendations in this newsletter series.
    • If not, then go back and fix the code.
    • Manual testing gives a false sense of security and leaves too many gaps — and it destroys the ability to edit and build independently.
  • Special-Purpose Production Workspaces.
    • You will need to add these eventually. Some contract or performance need will require your to segment customers, manage locations, or do something else that makes some Production environments different than others.
    • But first solve the general case, then add exceptions only after the general case is flowing smoothly.