Skip to main content

How to configure credentials

This guide will explain how to configure your great_expectations.yml project config to populate credentials from either a YAML file or a secret manager.

If your Great Expectations deployment is in an environment without a file system, refer to How to instantiate a Data Context without a yml file for credential configuration examples.

Prerequisites: This how-to guide assumes you have:

Steps

  1. Decide where you would like to save the desired credentials or config values - in a YAML file, environment variables, or a combination - then save the values. In most cases, we suggest using a config variables YAML file. YAML files make variables more visible, easily editable, and allow for modularization (e.g. one file for dev, another for prod).

    note
    • In the great_expectations.yml config file, environment variables take precedence over variables defined in a config variables YAML
    • Environment variable substitution is supported in both the great_expectations.yml and config variables config_variables.yml config file.

    If using a YAML file, save desired credentials or config values to great_expectations/uncommitted/config_variables.yml or another YAML file of your choosing:

    my_postgres_db_yaml_creds:  drivername: postgresql  host: localhost  port: 5432  username: postgres  password: ${MY_DB_PW}  database: postgres
    note
    • If you wish to store values that include the dollar sign character $, please escape them using a backslash \ so substitution is not attempted. For example in the above example for Postgres credentials you could set password: pa\$sword if your password is pa$sword. Say that 5 times fast, and also please choose a more secure password!
    • When you save values via the CLI, they are automatically escaped if they contain the $ character.
    • You can also have multiple substitutions for the same item, e.g. database_string: ${USER}:${PASSWORD}@${HOST}:${PORT}/${DATABASE}

    If using environment variables, set values by entering export ENV_VAR_NAME=env_var_value in the terminal or adding the commands to your ~/.bashrc file:

    export POSTGRES_DRIVERNAME=postgresqlexport POSTGRES_HOST=localhostexport POSTGRES_PORT=5432export POSTGRES_USERNAME=postgresexport POSTGRES_PW=export POSTGRES_DB=postgresexport MY_DB_PW=password

  2. If using a YAML file, set the config_variables_file_path key in your great_expectations.yml or leave the default.

    config_variables_file_path: uncommitted/config_variables.yml

  3. Replace credentials or other values in your great_expectations.yml with ${}-wrapped variable names (i.e. ${ENVIRONMENT_VARIABLE} or ${YAML_KEY}).

    datasources:  my_postgres_db:    class_name: Datasource    module_name: great_expectations.datasource    execution_engine:      module_name: great_expectations.execution_engine      class_name: SqlAlchemyExecutionEngine      credentials: ${my_postgres_db_yaml_creds}    data_connectors:      default_inferred_data_connector_name:        class_name: InferredAssetSqlDataConnector  my_other_postgres_db:    class_name: Datasource    module_name: great_expectations.datasource    execution_engine:      module_name: great_expectations.execution_engine      class_name: SqlAlchemyExecutionEngine      credentials:        drivername: ${POSTGRES_DRIVERNAME}        host: ${POSTGRES_HOST}        port: ${POSTGRES_PORT}        username: ${POSTGRES_USERNAME}        password: ${POSTGRES_PW}        database: ${POSTGRES_DB}    data_connectors:      default_inferred_data_connector_name:        class_name: InferredAssetSqlDataConnector

Additional Notes

  • The default config_variables.yml file located at great_expectations/uncommitted/config_variables.yml applies to deployments created using great_expectations init.
  • To view the full script used in this page, see it on GitHub: how_to_configure_credentials.py