Calvinverse - An example build infrastructure
This post introduces the Calvinverse project which provides the source code for the different resources required to create the infrastructure for a build pipeline. The Calvinverse resources have been developed for two main reasons:
- To provide me with a way to experiment with and learn more about immutable infrastructure and infrastructure-as-code as applied to build infrastructure.
- To provide resources that can be used to set up the infrastructure for a complete on-prem build system. The system should provide a build controller with build agents, artefact storage and all the necessary tools to monitor the different services and diagnose issues.
The Calvinverse resources can be configured and deployed for different sizes of infrastructure, from small setups only used by a few developers to a large setup used by many developers for the development of many products. How to configure the resources for small, medium or large environments and their hardware requirements will be discussed in future posts.
The resources in the Calvinverse project are designed to be run as a self-contained system. While daily maintenance is minimal it is not a hosted system so some maintenance is required. For instance OS updates will be required on a regular basis. These can either be applied to existing resources, through the automatic updates, or by applying the new updates to the templates and then replacing the existing resources with a new instance. The latter approach case can be automated, however there is no code in any of the Calvinverse repositories to do this automatically.
The different resources in the Calvinverse project contain a set of tools and applications which provide all the necessary capabilities to create the infrastructure for a build pipeline. Amongst these capabilities are service discovery, build execution, artefact storage, metrics, alerting and log processing.
The following applications and approaches are used for service discovery and configuration storage:
- Using Consul to provide service discovery and machine discovery via DNS inside an environment. An environment is defined as all machines that are part of a consul datacenter. It is possible to have multiple environments where the machines may all be on the same network but in general will not be communicating across environments. This is useful for cases where having multiple environments makes sense, for instance when having a production environment and a test environment. The benefit of using Consul as a DNS is that it allows a resource to have a consistent name across different environments without the DNS names clashing. For instance if there is a production environment and a test environment then it is possible to use the same DNS name for a resource, even though the actual machine names will be different. This allows using the Consul DNS name in tools and scripts without having to keep in mind the environment the tool is deployed in. Finally Consul is also used for the distributed key-value store that all applications can obtain configuration information from thereby centralizing the configuration information.
- Using one or more Vault instance to handle all the secrets required for the environment. Vault provides authenticated access for resources to securely access secrets, login credentials and other information that should be kept secure. This allows centralizing the storage and distribution of secrets
For the build work Calvinverse uses the following applications:
- Jenkins is used as the build controller.
- Build executors connect to Jenkins using the swarm plugin so that agents can connect when it starts. In the Calvinverse project there are currently only Windows based executors.
For artefact storage Calvin verse uses the Nexus application. The image is configured such that a new instance of the image will create artefact repositories for NuGet, Npm, Docker and general ZIP artefacts.
For message distribution Calvinverse uses the RabbitMQ application. The image is configured such that a new instance of the image will try to connect to the existing cluster in the environment. If no cluster exists then the first instance of the image will form the start of the cluster in the environment.
Metrics, monitoring and alerting capabilities are provided by the Influx stack, consisting of:
- InfluxDB for metrics collection.
- Grafana for dashboards.
- Telegraf is installed on each resource to collect metrics and send them to InfluxDb.
- Chronograf for alert configurations.
- Kapacitor for alerting.
Build and system logs are processed by the Elastic stack consisting off:
- Elasticsearch for log storage, both system logs and build logs
- Kibana for log dashboards.
- Logs collected via syslog-ng on Linux and a modified version of Filebeat on windows. Logs are sent to RabbitMQ to ensure that the unprocessed logs aren't lost when something any part of the log stack goes offline.
- Logstash for processing logs from RabbitMQ to Elasticsearch using filters
It should be noted that while the Calvinverse resources combine to create a complete build environment the resources might need some alterations to fit in with the work flow and processes that are being followed. After all each company is different and applies different workflows. Additionally users might want to replace some of the resources with versions of their own, e.g. to replace Influx with Prometheus.