Eine Einführung in Vault

1. Übersicht

In diesem Tutorial werden wir uns mit Hashicorp's Vault befassen - einem beliebten Tool zum sicheren Verwalten vertraulicher Informationen in modernen Anwendungsarchitekturen .

Zu den Hauptthemen, die wir behandeln werden, gehören:

  • Welches Problem versucht Vault zu lösen?
  • Vault's Architektur und Hauptkonzepte
  • Einrichtung einer einfachen Testumgebung
  • Interaktion mit Vault über das Befehlszeilentool

2. Das Problem mit sensiblen Informationen

Bevor wir uns mit Vault befassen, versuchen wir, das Problem zu verstehen, das es zu lösen versucht: das Management vertraulicher Informationen.

Die meisten Anwendungen benötigen Zugriff auf vertrauliche Daten, um ordnungsgemäß zu funktionieren . Beispielsweise kann für eine E-Commerce-Anwendung irgendwo ein Benutzername / Kennwort konfiguriert sein, um eine Verbindung zu ihrer Datenbank herzustellen. Möglicherweise sind auch API-Schlüssel erforderlich, um in andere Dienstanbieter wie Zahlungsgateways, Logistik und andere Geschäftspartner integriert zu werden.

Datenbankanmeldeinformationen und API-Schlüssel sind einige Beispiele für vertrauliche Informationen, die wir speichern und unseren Anwendungen auf sichere Weise zur Verfügung stellen müssen.

Eine einfache Lösung besteht darin, diese Anmeldeinformationen in einer Konfigurationsdatei zu speichern und beim Start zu lesen. Das Problem bei diesem Ansatz liegt jedoch auf der Hand. Wer Zugriff auf diese Datei hat, hat dieselben Datenbankrechte wie unsere Anwendung - normalerweise erhält er vollen Zugriff auf alle gespeicherten Daten.

Wir können versuchen, die Dinge etwas schwieriger zu machen, indem wir diese Dateien verschlüsseln. Dieser Ansatz wird jedoch nicht viel zur Gesamtsicherheit beitragen. Hauptsächlich, weil unsere Anwendung Zugriff auf den Hauptschlüssel haben muss. Wenn die Verschlüsselung auf diese Weise verwendet wird, wird nur ein „falsches“ Sicherheitsgefühl erzielt.

Moderne Anwendungen und Cloud-Umgebungen führen zu einer zusätzlichen Komplexität: Verteilte Dienste, mehrere Datenbanken, Messagingsysteme usw. weisen vertrauliche Informationen auf, die sich überall verteilen, wodurch das Risiko einer Sicherheitsverletzung erhöht wird.

Also was können wir tun? Lass es uns vaultieren!

3. Was ist Tresor?

Hashicorp Vault befasst sich mit dem Problem der Verwaltung vertraulicher Informationen - ein Geheimnis in der Sprache von Vault. "Verwalten" bedeutet in diesem Zusammenhang, dass Vault alle Aspekte einer vertraulichen Information kontrolliert : deren Generierung, Speicherung, Verwendung und nicht zuletzt deren Widerruf.

Hashicorp bietet zwei Versionen von Vault an. Die in diesem Artikel verwendete Open-Source-Version kann auch in kommerziellen Umgebungen kostenlos verwendet werden. Es ist auch eine kostenpflichtige Version verfügbar, die technischen Support bei verschiedenen SLAs und zusätzliche Funktionen wie HSM-Unterstützung (Hardware Security Module) umfasst.

3.1. Architektur & Hauptmerkmale

Die Architektur von Vault ist täuschend einfach. Seine Hauptkomponenten sind:

  • Ein Persistenz-Backend - Speicher für alle Geheimnisse
  • Ein API-Server, der Clientanforderungen verarbeitet und Operationen an Geheimnissen ausführt
  • Eine Reihe von geheimen Engines, eine für jeden Typ des unterstützten geheimen Typs

Indem wir die gesamte geheime Behandlung an Vault delegieren, können wir einige Sicherheitsprobleme mindern:

  • Unsere Anwendungen müssen sie nicht mehr speichern - fragen Sie Vault bei Bedarf und verwerfen Sie sie
  • Wir können kurzlebige Geheimnisse verwenden und so das „Zeitfenster“ einschränken, in dem ein Angreifer ein gestohlenes Geheimnis verwenden kann

Vault verschlüsselt alle Daten mit einem Verschlüsselungsschlüssel, bevor sie in den Speicher geschrieben werden. Dieser Verschlüsselungsschlüssel wird mit einem weiteren Schlüssel verschlüsselt - dem Hauptschlüssel, der nur beim Start verwendet wird.

Ein wichtiger Punkt bei der Implementierung von Vault ist, dass der Hauptschlüssel nicht auf dem Server gespeichert wird. Dies bedeutet, dass nicht einmal Vault nach dem Start auf die gespeicherten Daten zugreifen kann. Zu diesem Zeitpunkt befindet sich eine Vault-Instanz in einem "versiegelten" Zustand.

Später werden wir die Schritte ausführen, die zum Generieren des Hauptschlüssels und zum Entsiegeln einer Vault-Instanz erforderlich sind.

Nach dem Entsiegeln ist Vault bereit, API-Anforderungen zu akzeptieren. Diese Anforderungen müssen natürlich authentifiziert werden. Dadurch erfahren wir, wie Vault Clients authentifiziert und entscheidet, was sie können oder nicht.

3.2. Authentifizierung

Um auf Geheimnisse in Vault zugreifen zu können, muss sich ein Client mit einer der unterstützten Methoden authentifizieren . Die einfachste Methode verwendet Tokens, bei denen es sich lediglich um Zeichenfolgen handelt, die bei jeder API-Anforderung mithilfe eines speziellen HTTP-Headers gesendet werden.

Bei der Erstinstallation generiert Vault automatisch ein "Root-Token". Dieses Token entspricht dem Root-Superuser in Linux-Systemen, daher sollte seine Verwendung auf ein Minimum beschränkt werden. Als bewährte Methode sollten wir dieses Root-Token verwenden, um andere Token mit weniger Berechtigungen zu erstellen und es dann zu widerrufen. Dies ist jedoch kein Problem, da wir später ein weiteres Root-Token mithilfe von nicht versiegelten Schlüsseln generieren können.

Vault unterstützt auch andere Authentifizierungsmechanismen wie LDAP-, JWT- und TLS-Zertifikate. Alle diese Mechanismen bauen auf dem grundlegenden Token-Mechanismus auf: Sobald Vault unseren Client validiert, stellt es ein Token bereit, mit dem wir auf andere APIs zugreifen können.

Tokens sind einige Eigenschaften zugeordnet. Die Haupteigenschaften sind:

  • Eine Reihe zugehöriger Richtlinien (siehe nächster Abschnitt)
  • Zeit zu leben
  • Ob es erneuert werden kann
  • Maximale Nutzungsanzahl

Sofern nicht anders angegeben, bilden von Vault erstellte Token eine Eltern-Kind-Beziehung. Ein untergeordnetes Token kann höchstens die gleichen Berechtigungen haben wie das übergeordnete Token.

Das Gegenteil ist nicht der Fall: Wir können - und tun dies normalerweise - ein untergeordnetes Token mit restriktiven Richtlinien erstellen. Ein weiterer wichtiger Punkt in dieser Beziehung: Wenn wir ein Token ungültig machen, werden auch alle untergeordneten Token und ihre Nachkommen ungültig .

3.3. Richtlinien

Richtlinien definieren genau, auf welche Geheimnisse ein Client zugreifen kann und welche Vorgänge er mit ihnen ausführen kann . Mal sehen, wie eine einfache Richtlinie aussieht:

path "secret/accounting" { capabilities = [ "read" ] }

Hier haben wir die HCL-Syntax (Hashicorp's Configuration Language) verwendet, um unsere Richtlinie zu definieren. Vault unterstützt zu diesem Zweck auch JSON, aber wir werden uns in unseren Beispielen an HCL halten, da es einfacher zu lesen ist.

Policies in Vault are “deny by default”. A token attached to this sample policy will get access to secrets stored under secret/accounting and nothing else. At creation time a token can be attached to multiple policies. This is very useful because it allows us to create and test smaller policies and then apply them as required.

Another important aspect of policies is that they leverage lazy-evaluation. This means that we can update a given policy and all tokens will be affected immediately.

The policies described so far are also called Access Control List Policies, or ACL Policies. Vault also supports two additional policy types: EGP and RGP policies. Those are only available in the paid versions and extend the basic policy syntax with Sentinel support.

When available, this allows us to take into account in our policies additional attributes such as time of the day, multiple authentication factors, client network origin, and so on. For instance, we can define a policy that allows access to a given secret only on business hours.

We can find more details on the policy syntax in Vault's documentation.

4. Secret Types

Vault support a range of different secret types which address different use cases:

  • Key-Value: simple static key-values pairs
  • Dynamically generated credentials: generated by Vault upon request by a client
  • Cryptographic keys: Used to perform cryptographic functions with client data

Each secret type is defined by the following attributes:

  • A mountpoint, which defines its REST API prefix
  • A set of operations exposed through the corresponding API
  • A set of configuration parameters

A given secret instance is accessible via a path, much like a directory tree in a file system. The first component of this path corresponds to the mount point where all secrets of this type are located.

For instance, the string secret/my-application corresponds to the path under which we can find key-value pairs for my-application.

4.1. Key-Value Secrets

Key-Value secrets are, as the name implies, simple pairs in the available under a given path. For instance, we can store the pair foo=bar under the path /secret/my-application.

Later on, we use the same path to retrieve the same pair or pairs – multiple pairs can be stored under the same path.

Vault support three kinds of Key-Value secrets:

  • Non-versioned Key-Pairs, where updates replace existing values
  • Versioned Key-Pairs, which keep up to a configurable number of old versions
  • Cubbyhole, a special type of non-versioned key-pairs whose values are scoped to a given access token (more on those later).

Key-Value secrets are static by nature, so there is no concept of an associated expiration associated with them. The main use case for this kind of secret is to store credentials to access external systems, such as API keys.

In such scenarios credential updates are a semi-manual process, usually requiring someone to acquire new credentials and using Vault's command line or its UI to enter the new values.

4.2. Dynamically Generated Secrets

Dynamic secrets are generated on the fly by Vault when requested by an application. Vault support several types of dynamic secrets, including the following ones:

  • Database credentials
  • SSH Key Pairs
  • X.509 Certificates
  • AWS Credentials
  • Google Cloud service accounts
  • Active Directory accounts

All these follow the same usage pattern. First, we configure the secret engine with the details required to connect to the associated service. Then, we define one or more roles, which describe the actual secret creation.

Let's take the Database secret engine as an example. First, we must configure Vault with all user database connections details, including credentials from a preexisting user with admin privileges to create new users.

Then we create one or more roles (Vault roles, not Database roles) containing the actual SQL statements used to create a new user. Those usually include not only the user creation statement but also all the required grant statements required to access schema objects (tables, views and so on).

When a client accesses the corresponding API, Vault will create a new temporary user in the database using the provided statements and return its credentials. The client can then use those credentials to access the database during the period defined by the time-to-live attribute of the requested role.

Once a credential reaches its expiration time, Vault will automatically revoke any privilege associated with this user. A client can also request Vault to renew those credentials. The renewal process will happen only if supported by the specific database driver and allowed by the associated policy.

4.3. Cryptographic Keys

Secret engines of type handle cryptographic functions such as encryption, decryption, signing and so on. All those operations use cryptographic keys generated and stored internally by Vault. Unless explicitly told to do so, Vault will never expose a given cryptographic key.

The associated API allows clients to send Vault plain-text data and receive an encrypted version of it. The opposite is also possible: We can send encrypted data and get back the original text.

Currently, there is only one engine of this type: the Transit engine. This engine supports popular keys types, such as RSA and ECDSA, and also supports Convergent Encryption. When using this mode, a given plaintext value always result in the same cyphertext result, a property that is very useful in some applications.

For instance, we can use this mode to encrypt credit card numbers in a transaction log table. With convergent encryption, every time we insert a new transaction, the encrypted credit card value would be the same, thus allowing the use of regular SQL queries for reporting, searching and so on.

5. Vault Setup

In this section, we will create a local test environment so we test the Vault's capabilities.

Vault's deployment is simple: just download the package that corresponds to our operating system and extracts its executable (vault or vault.exe on Windows) to some directory on our PATH. This executable contains the server and is also the standard client. There is also an official Docker image available, but we will not cover it here.

Vault support a development mode, which is fine for some quick testing and getting used to its command line tool, but it is way too simplistic for real use cases: all data is lost on restart and API access uses plain HTTP.

Instead, we'll use file-based persistent storage and setup HTTPS so we can explore some of the real-life configuration details that can be a source of problems.

5.1. Starting Vault Server

Vault uses a configuration file using HCL or JSON format. The following file defines all the configuration needed to start our server using a file storage and a self-signed certificate:

storage "file" { path = "./vault-data" } listener "tcp" { address = "127.0.0.1:8200" tls_cert_file = "./src/test/vault-config/localhost.cert" tls_key_file = "./src/test/vault-config/localhost.key" }

Now, let's run Vault. Open a command shell, go to the directory containing our configuration file and run this command:

$ vault server -config ./vault-test.hcl

Vault will start and show a few initialization messages. They'll include its version, some configuration details and the address where the API is available. That's it – our Vault server is up and running.

5.2. Vault Initialization

Our Vault server now is running, but since this is its first run, we need to initialize it.

Let's open a new shell and execute the following commands to achieve this:

$ export VAULT_ADDR=//localhost:8200 $ export VAULT_CACERT=./src/test/vault-config/localhost.cert $ vault operator init

Here we have defined a few environment variables, so we don't have to pass them to Vault every time as parameters:

  • VAULT_ADDR: base URI where our API server will serve requests
  • VAULT_CACERT: Path to our server's certificate public key

In our case, we use the VAULT_CACERT so we can use HTTPS to access Vault's API. We need this because we're using self-signed certificates. This would not be necessary for productions environments, where we usually have access to CA-signed certificates.

After issuing the above command, we should see a message like this:

Unseal Key 1:  Unseal Key 2:  Unseal Key 3:  Unseal Key 4:  Unseal Key 5:  Initial Root Token:  ... more messages omitted

The five first lines are the master key shares that we will later use to unseal Vault's storage. Please note that Vault only displays the master key shares will during initialization – and never more.Take note and store them safely or we'll lose access to our secrets upon server restart!

Also, please take note of the root token, as we will need it later. Unlike unseal keys, root tokens can easily be generated at a later time, so it is safe to destroy it once all configuration tasks are complete. Since we will be issuing commands later that require an authentication token, let's save the root token for now in an environment variable:

$ export VAULT_TOKEN= (Unix/Linux)

Let's see our server status now that we have initialized it, with the following command:

$ vault status Key Value --- ----- Seal Type shamir Sealed true Total Shares 5 Threshold 3 Unseal Progress 0/3 Unseal Nonce n/a Version 0.10.4 HA Enabled false

We can see that Vault is still sealed. We can also follow the unseal progress: “0/3” means that Vault needs three shares, but got none so far. Let's move ahead and provide it with our shares.

5.3. Vault Unseal

We now unseal Vault so we can start using its secret services. We need to provide any three of the five key shares in order to complete the unseal process:

$ vault operator unseal  $ vault operator unseal  $ vault operator unseal 

After issuing each command vault will print the unseal progress, including how many shares it needs. Upon sending the last key share, we'll see a message like this:

Key Value --- ----- Seal Type shamir Sealed false ... other properties omitted

The “Sealed” property is “false” in this case, which means that Vault is ready to accept commands.

6. Testing Vault

In this section, we will test our Vault setup using two of its supported secret types: Key/Value and Database. We will also show how to create new tokens with specific policies attached to them.

6.1. Using Key/Value Secrets

First, let's store secret Key-Value pairs and read them back. Assuming the command shell used to initialize Vault is still open, we use the following command to store those pairs under the secret/fakebank path:

$ vault kv put secret/fakebank api_key=abc1234 api_secret=1a2b3c4d

We can now recover those pairs at any time with the following command:

$ vault kv get secret/fakebank ======= Data ======= Key Value --- ----- api_key abc1234 api_secret 1a2b3c4d 

This simple test shows us that Vault is working as it should. We can now test some additional functionalities.

6.2. Creating New Tokens

So far we have used the root token in order to authenticate our requests. Since a root token is way too powerful, it is considered a best practice to use tokens with fewer privileges and shorter time-to-live.

Let's create a new token that we can use just like the root token, but expires after just a minute:

$ vault token create -ttl 1m Key Value --- ----- token  token_accessor  token_duration 1m token_renewable true token_policies ["root"] identity_policies [] policies ["root"]

Let's test this token, using it to read the key/value pairs that we've created before:

$ export VAULT_TOKEN= $ vault kv get secret/fakebank ======= Data ======= Key Value --- ----- api_key abc1234 api_secret 1a2b3c4d

If we wait a minute and try to reissue this command, we get an error message:

$ vault kv get secret/fakebank Error making API request. URL: GET //localhost:8200/v1/sys/internal/ui/mounts/secret/fakebank Code: 403. Errors: * permission denied

The message indicates that our token is no longer valid, which is what we've expected.

6.3. Testing Policies

The sample token we've created in the previous section was shorted lived, but still very powerful. Let's now use policies to create more restricted tokens.

For instance, let's define a policy that allows only read access to the secret/fakebank path we used before:

$ cat > sample-policy.hcl <
    

Now we create a token with this policy with the following command:

$ export VAULT_TOKEN= $ vault token create -policy=fakebank-ro Key Value --- ----- token token_accessor token_duration 768h token_renewable true token_policies ["default" "fakebank-ro"] identity_policies [] policies ["default" "fakebank-ro"]

As we've done before, let's read our secret values using this token:

$ export VAULT_TOKEN= $ vault kv get secret/fakebank ======= Data ======= Key Value --- ----- api_key abc1234 api_secret 1a2b3c4d

So far, so good. We can read data, as expected. Let's see what happens when we try to update this secret:

$ vault kv put secret/fakebank api_key=foo api_secret=bar Error writing data to secret/fakebank: Error making API request. URL: PUT //127.0.0.1:8200/v1/secret/fakebank Code: 403. Errors: * permission denied

Since our policy does not explicitly allows writes, Vault returns a 403 – Access Denied status code.

6.4. Using Dynamic Database Credentials

As our final example in this article, let's use Vault's Database secret engine in order to create dynamic credentials. We assume here that we have a MySQL server available locally and that we can access it with “root” privileges. We will also use a very simple schema consisting of a single table – account .

The SQL script used to create this schema and the privileged user is available here.

Now, let's configure Vault to use this database. The database secret engine is not enabled by default, so we must fix this before we can proceed:

$ vault secrets enable database Success! Enabled the database secrets engine at: database/

We now create a database configuration resource :

$ vault write database/config/mysql-fakebank \ plugin_name=mysql-legacy-database-plugin \ connection_url="{{username}}:{{password}}@tcp(127.0.0.1:3306)/fakebank" \ allowed_roles="*" \ username="fakebank-admin" \ password="Sup&rSecre7!"

The path prefix database/config is where all database configurations must be stored. We choose the name mysql-fakebank so we can easily figure out to which database this configuration refers to. As for the configuration keys:

  • plugin_name: Defines which database plugin will be used. The available plugin names are described in Vault's docs
  • connection_url: This is a template used by the plugin when connecting to the database. Notice the {{username}} and {{password}} template placeholders. When connecting to the database, Vault will replace those placeholders by actual values
  • allowed_roles: Define which Vault roles (discussed next) can use this configuration. In our case we use “*”, so its available to all roles
  • username & password: This is the account that Vault will use to perform database operations, such as creating a new user and revoking its privileges

Vault Database Role Setup

The final configuration task is to create a Vault database role resource that contains the SQL commands required to create a user. We can create as many roles as needed, according to our security requirements.

Here, we create a role that grants read-only access to all tables of the fakebank schema:

$ vault write database/roles/fakebank-accounts-ro \ db_name=mysql-fakebank \ creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT ON fakebank.* TO '{{name}}'@'%';" 

The database engine defines the path prefix database/roles as the location to store roles. fakebank-accounts-ro is the role name that we'll later use when creating dynamic credentials. We also supply the following keys:

  • db_name: Name of an existing database configuration. Corresponds to the last part of the path we used when creating the configuration resource
  • creation_statements: A list of SQL statement templates that Vault will use to create a new user

Creating Dynamic Credentials

Once we have a database role and its corresponding configuration ready, we generate new dynamic credentials with the following command:

$ vault read database/creds/fakebank-accounts-ro Key Value --- ----- lease_id database/creds/fakebank-accounts-ro/0c0a8bef-761a-2ef2-2fed-4ee4a4a076e4 lease_duration 1h lease_renewable true password username 

The database/creds prefix is used to generate credentials for the available roles. Since we have used the fakebank-accounts-ro role, the returned username/password will be restricted to select operations.

We can verify this by connecting to the database using the supplied credentials and then performing some SQL commands:

$ mysql -h 127.0.0.1 -u -p fakebank Enter password: MySQL [fakebank]> select * from account; ... omitted for brevity 2 rows in set (0.00 sec) MySQL [fakebank]> delete from account; ERROR 1142 (42000): DELETE command denied to user 'v-fake-9xoSKPkj1'@'localhost' for table 'account' 

We can see that the first select completed successfully, but we could not perform the delete statement. Finally, if we wait for one hour and try to connect using those same credentials, we will not be able to connect anymore to the database. Vault has automatically revoked all privileges from this user

7. Conclusion

In this article have explored the basics of Hashicorp's Vault, including some background on the problem it tries to address, its architecture and basic use.

Along the way, we have created a simple but functional test environment that we´ll use in follow-up articles.

The next article will cover a very specific use case for Vault: Using it in the context of Spring Boot application. Stay tuned!