guide/go_decoding_hcldec.rst - hashicorp/hcl/v2 - Git at Google

 .. go:package:: hcldec

 .. _go-decoding-hcldec:

 Decoding With Dynamic Schema
 ============================

 In section :ref:`go-decoding-gohcl`, we saw the most straightforward way to
 access the content from an HCL file, decoding directly into a Go value whose
 type is known at application compile time.

 For some applications, it is not possible to know the schema of the entire
 configuration when the application is built. For example, `HashiCorp Terraform`_
 uses HCL as the foundation of its configuration language, but parts of the
 configuration are handled by plugins loaded dynamically at runtime, and so
 the schemas for these portions cannot be encoded directly in the Terraform
 source code.

 HCL's ``hcldec`` package offers a different approach to decoding that allows
 schemas to be created at runtime, and the result to be decoded into
 dynamically-typed data structures.

 The sections below are an overview of the main parts of package ``hcldec``.
 For full details, see
 `the package godoc <https://godoc.org/github.com/hashicorp/hcl/v2/hcldec>`_.

 .. _`HashiCorp Terraform`: https://www.terraform.io/

 Decoder Specification
 ---------------------

 Whereas :go:pkg:`gohcl` infers the expected schema by using reflection against
 the given value, ``hcldec`` obtains schema through a decoding *specification*,
 which is a set of instructions for mapping HCL constructs onto a dynamic
 data structure.

 The ``hcldec`` package contains a number of different specifications, each
 implementing :go:type:`hcldec.Spec` and having a ``Spec`` suffix on its name.
 Each spec has two distinct functions:

 * Adding zero or more validation constraints on the input configuration file.

 * Producing a result value based on some elements from the input file.

 The most common pattern is for the top-level spec to be a
 :go:type:`hcldec.ObjectSpec` with nested specifications defining either blocks
 or attributes, depending on whether the configuration file will be
 block-structured or flat.

 .. code-block:: go

   spec := hcldec.ObjectSpec{
       "io_mode": &hcldec.AttrSpec{
           Name: "io_mode",
           Type: cty.String,
       },
       "services": &hcldec.BlockMapSpec{
           TypeName:   "service",
           LabelNames: []string{"type", "name"},
           Nested:     hcldec.ObjectSpec{
               "listen_addr": &hcldec.AttrSpec{
                   Name:     "listen_addr",
                   Type:     cty.String,
                   Required: true,
               },
               "processes": &hcldec.BlockMapSpec{
                   TypeName:   "service",
                   LabelNames: []string{"name"},
                   Nested:     hcldec.ObjectSpec{
                       "command": &hcldec.AttrSpec{
                           Name:     "command",
                           Type:     cty.List(cty.String),
                           Required: true,
                       },
                   },
               },
           },
       },
   }
   val, moreDiags := hcldec.Decode(f.Body, spec, nil)
   diags = append(diags, moreDiags...)

 The above specification expects a configuration shaped like our example in
 :ref:`intro`, and calls for it to be decoded into a dynamic data structure
 that would have the following shape if serialized to JSON:

 .. code-block:: JSON

   {
     "io_mode": "async",
     "services": {
       "http": {
         "web_proxy": {
           "listen_addr": "127.0.0.1:8080",
           "processes": {
             "main": {
               "command": ["/usr/local/bin/awesome-app", "server"]
             },
             "mgmt": {
               "command": ["/usr/local/bin/awesome-app", "mgmt"]
             }
           }
         }
       }
     }
   }

 .. go:package:: cty

 Types and Values With ``cty``
 -----------------------------

 HCL's expression interpreter is implemented in terms of another library called
 :go:pkg:`cty`, which provides a type system which HCL builds on and a robust
 representation of dynamic values in that type system. You could think of
 :go:pkg:`cty` as being a bit like Go's own :go:pkg:`reflect`, but for the
 results of HCL expressions rather than Go programs.

 The full details of this system can be found in
 `its own repository <https://github.com/zclconf/go-cty>`_, but this section
 will cover the most important highlights, because ``hcldec`` specifications
 include :go:pkg:`cty` types (as seen in the above example) and its results are
 :go:pkg:`cty` values.

 ``hcldec`` works directly with :go:pkg:`cty` — as opposed to converting values
 directly into Go native types — because the functionality of the :go:pkg:`cty`
 packages then allows further processing of those values without any loss of
 fidelity or range. For example, :go:pkg:`cty` defines a JSON encoding of its
 values that can be decoded losslessly as long as both sides agree on the value
 type that is expected, which is a useful capability in systems where some sort
 of RPC barrier separates the main program from its plugins.

 Types are instances of :go:type:`cty.Type`, and are constructed from functions
 and variables in :go:pkg:`cty` as shown in the above example, where the string
 attributes are typed as ``cty.String``, which is a primitive type, and the list
 attribute is typed as ``cty.List(cty.String)``, which constructs a new list
 type with string elements.

 Values are instances of :go:type:`cty.Value`, and can also be constructed from
 functions in :go:pkg:`cty`, using the functions that include ``Val`` in their
 names or using the operation methods available on :go:type:`cty.Value`.

 In most cases you will eventually want to use the resulting data as native Go
 types, to pass it to non-:go:pkg:`cty`-aware code. To do this, see the guides
 on
 `Converting between types <https://github.com/zclconf/go-cty/blob/master/docs/convert.md>`_
 (staying within :go:pkg:`cty`) and
 `Converting to and from native Go values <https://github.com/zclconf/go-cty/blob/master/docs/gocty.md>`_.

 Partial Decoding
 ----------------

 Because the ``hcldec`` result is always a value, the input is always entirely
 processed in a single call, unlike with :go:pkg:`gohcl`.

 However, both :go:pkg:`gohcl` and :go:pkg:`hcldec` take :go:type:`hcl.Body` as
 the representation of input, and so it is possible and common to mix them both
 in the same program.

 A common situation is that :go:pkg:`gohcl` is used in the main program to
 decode the top level of configuration, which then allows the main program to
 determine which plugins need to be loaded to process the leaf portions of
 configuration. In this case, the portions that will be interpreted by plugins
 are retained as opaque :go:type:`hcl.Body` until the plugins have been loaded,
 and then each plugin provides its :go:type:`hcldec.Spec` to allow decoding the
 plugin-specific configuration into a :go:type:`cty.Value` which be
 transmitted to the plugin for further processing.

 In our example from :ref:`intro`, perhaps each of the different service types
 is managed by a plugin, and so the main program would decode the block headers
 to learn which plugins are needed, but process the block bodies dynamically:

 .. code-block:: go

    type ServiceConfig struct {
      Type         string   `hcl:"type,label"`
      Name         string   `hcl:"name,label"`
      PluginConfig hcl.Body `hcl:",remain"`
    }
    type Config struct {
      IOMode   string          `hcl:"io_mode"`
      Services []ServiceConfig `hcl:"service,block"`
    }

    var c Config
    moreDiags := gohcl.DecodeBody(f.Body, nil, &c)
    diags = append(diags, moreDiags...)
    if moreDiags.HasErrors() {
        // (show diags in the UI)
        return
    }

    for _, sc := range c.Services {
        pluginName := block.Type

        // Totally-hypothetical plugin manager (not part of HCL)
        plugin, err := pluginMgr.GetPlugin(pluginName)
        if err != nil {
            diags = diags.Append(&hcl.Diagnostic{ /* ... */ })
            continue
        }
        spec := plugin.ConfigSpec() // returns hcldec.Spec

        // Decode the block body using the plugin's given specification
        configVal, moreDiags := hcldec.Decode(sc.PluginConfig, spec, nil)
        diags = append(diags, moreDiags...)
        if moreDiags.HasErrors() {
            continue
        }

        // Again, hypothetical API within your application itself, and not
        // part of HCL. Perhaps plugin system serializes configVal as JSON
        // and sends it over to the plugin.
        svc := plugin.NewService(configVal)
        serviceMgr.AddService(sc.Name, svc)
    }


 Variables and Functions
 -----------------------

 The final argument to ``hcldec.Decode`` is an expression evaluation context,
 just as with ``gohcl.DecodeBlock``.

 This object can be constructed using
 :ref:`the gohcl helper function <go-decoding-gohcl-evalcontext>` as before if desired, but
 you can also choose to work directly with :go:type:`hcl.EvalContext` as
 discussed in :ref:`go-expression-eval`:

 .. code-block:: go

    ctx := &hcl.EvalContext{
        Variables: map[string]cty.Value{
            "pid": cty.NumberIntVal(int64(os.Getpid())),
        },
    }
   val, moreDiags := hcldec.Decode(f.Body, spec, ctx)
   diags = append(diags, moreDiags...)

 As you can see, this lower-level API also uses :go:pkg:`cty`, so it can be
 particularly convenient in situations where the result of dynamically decoding
 one block must be available to expressions in another block.
	.. go:package:: hcldec

	.. _go-decoding-hcldec:

	Decoding With Dynamic Schema
	============================

	In section :ref:`go-decoding-gohcl`, we saw the most straightforward way to
	access the content from an HCL file, decoding directly into a Go value whose
	type is known at application compile time.

	For some applications, it is not possible to know the schema of the entire
	configuration when the application is built. For example, `HashiCorp Terraform`_
	uses HCL as the foundation of its configuration language, but parts of the
	configuration are handled by plugins loaded dynamically at runtime, and so
	the schemas for these portions cannot be encoded directly in the Terraform
	source code.

	HCL's ``hcldec`` package offers a different approach to decoding that allows
	schemas to be created at runtime, and the result to be decoded into
	dynamically-typed data structures.

	The sections below are an overview of the main parts of package ``hcldec``.
	For full details, see
	`the package godoc <https://godoc.org/github.com/hashicorp/hcl/v2/hcldec>`_.

	.. _`HashiCorp Terraform`: https://www.terraform.io/

	Decoder Specification
	---------------------

	Whereas :go:pkg:`gohcl` infers the expected schema by using reflection against
	the given value, ``hcldec`` obtains schema through a decoding specification,
	which is a set of instructions for mapping HCL constructs onto a dynamic
	data structure.

	The ``hcldec`` package contains a number of different specifications, each
	implementing :go:type:`hcldec.Spec` and having a ``Spec`` suffix on its name.
	Each spec has two distinct functions:

	* Adding zero or more validation constraints on the input configuration file.

	* Producing a result value based on some elements from the input file.

	The most common pattern is for the top-level spec to be a
	:go:type:`hcldec.ObjectSpec` with nested specifications defining either blocks
	or attributes, depending on whether the configuration file will be
	block-structured or flat.

	.. code-block:: go

	spec := hcldec.ObjectSpec{
	"io_mode": &hcldec.AttrSpec{
	Name: "io_mode",
	Type: cty.String,
	},
	"services": &hcldec.BlockMapSpec{
	TypeName: "service",
	LabelNames: []string{"type", "name"},
	Nested: hcldec.ObjectSpec{
	"listen_addr": &hcldec.AttrSpec{
	Name: "listen_addr",
	Type: cty.String,
	Required: true,
	},
	"processes": &hcldec.BlockMapSpec{
	TypeName: "service",
	LabelNames: []string{"name"},
	Nested: hcldec.ObjectSpec{
	"command": &hcldec.AttrSpec{
	Name: "command",
	Type: cty.List(cty.String),
	Required: true,
	},
	},
	},
	},
	},
	}
	val, moreDiags := hcldec.Decode(f.Body, spec, nil)
	diags = append(diags, moreDiags...)

	The above specification expects a configuration shaped like our example in
	:ref:`intro`, and calls for it to be decoded into a dynamic data structure
	that would have the following shape if serialized to JSON:

	.. code-block:: JSON

	{
	"io_mode": "async",
	"services": {
	"http": {
	"web_proxy": {
	"listen_addr": "127.0.0.1:8080",
	"processes": {
	"main": {
	"command": ["/usr/local/bin/awesome-app", "server"]
	},
	"mgmt": {
	"command": ["/usr/local/bin/awesome-app", "mgmt"]
	}
	}
	}
	}
	}
	}

	.. go:package:: cty

	Types and Values With ``cty``
	-----------------------------

	HCL's expression interpreter is implemented in terms of another library called
	:go:pkg:`cty`, which provides a type system which HCL builds on and a robust
	representation of dynamic values in that type system. You could think of
	:go:pkg:`cty` as being a bit like Go's own :go:pkg:`reflect`, but for the
	results of HCL expressions rather than Go programs.

	The full details of this system can be found in
	`its own repository <https://github.com/zclconf/go-cty>`_, but this section
	will cover the most important highlights, because ``hcldec`` specifications
	include :go:pkg:`cty` types (as seen in the above example) and its results are
	:go:pkg:`cty` values.

	``hcldec`` works directly with :go:pkg:`cty` — as opposed to converting values
	directly into Go native types — because the functionality of the :go:pkg:`cty`
	packages then allows further processing of those values without any loss of
	fidelity or range. For example, :go:pkg:`cty` defines a JSON encoding of its
	values that can be decoded losslessly as long as both sides agree on the value
	type that is expected, which is a useful capability in systems where some sort
	of RPC barrier separates the main program from its plugins.

	Types are instances of :go:type:`cty.Type`, and are constructed from functions
	and variables in :go:pkg:`cty` as shown in the above example, where the string
	attributes are typed as ``cty.String``, which is a primitive type, and the list
	attribute is typed as ``cty.List(cty.String)``, which constructs a new list
	type with string elements.

	Values are instances of :go:type:`cty.Value`, and can also be constructed from
	functions in :go:pkg:`cty`, using the functions that include ``Val`` in their
	names or using the operation methods available on :go:type:`cty.Value`.

	In most cases you will eventually want to use the resulting data as native Go
	types, to pass it to non-:go:pkg:`cty`-aware code. To do this, see the guides
	on
	`Converting between types <https://github.com/zclconf/go-cty/blob/master/docs/convert.md>`_
	(staying within :go:pkg:`cty`) and
	`Converting to and from native Go values <https://github.com/zclconf/go-cty/blob/master/docs/gocty.md>`_.

	Partial Decoding
	----------------

	Because the ``hcldec`` result is always a value, the input is always entirely
	processed in a single call, unlike with :go:pkg:`gohcl`.

	However, both :go:pkg:`gohcl` and :go:pkg:`hcldec` take :go:type:`hcl.Body` as
	the representation of input, and so it is possible and common to mix them both
	in the same program.

	A common situation is that :go:pkg:`gohcl` is used in the main program to
	decode the top level of configuration, which then allows the main program to
	determine which plugins need to be loaded to process the leaf portions of
	configuration. In this case, the portions that will be interpreted by plugins
	are retained as opaque :go:type:`hcl.Body` until the plugins have been loaded,
	and then each plugin provides its :go:type:`hcldec.Spec` to allow decoding the
	plugin-specific configuration into a :go:type:`cty.Value` which be
	transmitted to the plugin for further processing.

	In our example from :ref:`intro`, perhaps each of the different service types
	is managed by a plugin, and so the main program would decode the block headers
	to learn which plugins are needed, but process the block bodies dynamically:

	.. code-block:: go

	type ServiceConfig struct {
	Type string `hcl:"type,label"`
	Name string `hcl:"name,label"`
	PluginConfig hcl.Body `hcl:",remain"`
	}
	type Config struct {
	IOMode string `hcl:"io_mode"`
	Services []ServiceConfig `hcl:"service,block"`
	}

	var c Config
	moreDiags := gohcl.DecodeBody(f.Body, nil, &c)
	diags = append(diags, moreDiags...)
	if moreDiags.HasErrors() {
	// (show diags in the UI)
	return
	}

	for _, sc := range c.Services {
	pluginName := block.Type

	// Totally-hypothetical plugin manager (not part of HCL)
	plugin, err := pluginMgr.GetPlugin(pluginName)
	if err != nil {
	diags = diags.Append(&hcl.Diagnostic{ /* ... */ })
	continue
	}
	spec := plugin.ConfigSpec() // returns hcldec.Spec

	// Decode the block body using the plugin's given specification
	configVal, moreDiags := hcldec.Decode(sc.PluginConfig, spec, nil)
	diags = append(diags, moreDiags...)
	if moreDiags.HasErrors() {
	continue
	}

	// Again, hypothetical API within your application itself, and not
	// part of HCL. Perhaps plugin system serializes configVal as JSON
	// and sends it over to the plugin.
	svc := plugin.NewService(configVal)
	serviceMgr.AddService(sc.Name, svc)
	}


	Variables and Functions
	-----------------------

	The final argument to ``hcldec.Decode`` is an expression evaluation context,
	just as with ``gohcl.DecodeBlock``.

	This object can be constructed using
	:ref:`the gohcl helper function <go-decoding-gohcl-evalcontext>` as before if desired, but
	you can also choose to work directly with :go:type:`hcl.EvalContext` as
	discussed in :ref:`go-expression-eval`:

	.. code-block:: go

	ctx := &hcl.EvalContext{
	Variables: map[string]cty.Value{
	"pid": cty.NumberIntVal(int64(os.Getpid())),
	},
	}
	val, moreDiags := hcldec.Decode(f.Body, spec, ctx)
	diags = append(diags, moreDiags...)

	As you can see, this lower-level API also uses :go:pkg:`cty`, so it can be
	particularly convenient in situations where the result of dynamically decoding
	one block must be available to expressions in another block.