v1.16.5/docs/plugin-protocol/README.md

# Terraform Plugin Protocol

This directory contains documentation about the physical wire protocol that
Terraform Core uses to communicate with provider plugins.

Most providers are not written directly against this protocol. Instead, prefer
to use an SDK that implements this protocol and write the provider against
the SDK's API.

----

**If you want to write a plugin for Terraform, please refer to
[Plugin development](https://developer.hashicorp.com/terraform/plugin) instead.**

This documentation is for those who are developing _Terraform SDKs_, rather
than those implementing plugins.

----

From Terraform v0.12.0 onwards, Terraform's plugin protocol is built on
[gRPC](https://grpc.io/). This directory contains `.proto` definitions of
different versions of Terraform's protocol.

Only `.proto` files published as part of Terraform release tags are actually
official protocol versions. If you are reading this directory on the `main`
branch or any other development branch then it may contain protocol definitions
that are not yet finalized and that may change before final release.

## RPC Plugin Model

Terraform plugins are normal executable programs that, when launched, expose
gRPC services on a server accessed via the loopback interface. Terraform Core
discovers and launches plugins, waits for a handshake to be printed on the
plugin's `stdout`, and then connects to the indicated port number as a
gRPC client.

For this reason, we commonly refer to Terraform Core itself as the plugin
"client" and the plugin program itself as the plugin "server". Both of these
processes run locally, with the server process appearing as a child process
of the client. Terraform Core controls the lifecycle of these server processes
and will terminate them when they are no longer required.

The startup and handshake protocol is not currently documented. We hope to
document it here or to link to external documentation on it in future.

## Versioning Strategy

The Plugin Protocol uses a versioning strategy that aims to allow gradual
enhancements to the protocol while retaining compatibility, but also to allow
more significant breaking changes from time to time while allowing old and
new plugins to be used together for some period.

The versioning strategy described below was introduced with protocol version
5.0 in Terraform v0.12. Prior versions of Terraform and prior protocol versions
do not follow this strategy.

The authoritative definition for each protocol version is in this directory
as a Protocol Buffers (protobuf) service definition. The files follow the
naming pattern `tfpluginX.proto`, where X is the major version.

### Major and minor versioning

The minor version increases for each change introducing optional new
functionality that can be ignored by implementations of prior versions. For
example, if a new field were added to an response message, it could be a minor
release as long as Terraform Core can provide some default behavior when that
field is not populated.

The major version increases for any significant change to the protocol where
compatibility is broken. However, Terraform Core and an SDK may both choose
to support multiple major versions at once: the plugin handshake includes a
negotiation step where client and server can work together to select a
mutually-supported major version.

The major version number is encoded into the protobuf package name: major
version 5 uses the package name `tfplugin5`, and one day major version 6
will switch to `tfplugin6`. This change of name allows a plugin server to
implement multiple major versions at once, by exporting multiple gRPC services.
Minor version differences rely instead on feature-detection mechanisms, so they
are not represented directly on the wire and exist primarily as a human
communication tool to help us easily talk about which software supports which
features.

## Version compatibility for Core, SDK, and Providers

A particular version of Terraform Core has both a minimum minor version it
requires and a maximum major version that it supports. A particular version of
Terraform Core may also be able to optionally use a newer minor version when
available, but fall back on older behavior when that functionality is not
available.

Likewise, each provider plugin release is compatible with a set of versions.
The compatible versions for a provider are a list of major and minor version
pairs, such as "4.0", "5.2", which indicates that the provider supports the
baseline features of major version 4 and supports major version 5 including
the enhancements from both minor versions 1 and 2. This provider would
therefore be compatible with a Terraform Core release that supports only
protocol version 5.0, since major version 5 is supported and the optional
5.1 and 5.2 enhancements will be ignored.

If Terraform Core and the plugin do not have at least one mutually-supported
major version, Terraform Core will return an error from `terraform init`
during plugin installation:

```
Provider "aws" v1.0.0 is not compatible with Terraform v0.12.0.

Provider version v2.0.0 is the earliest compatible version.
Select it with the following version constraint:

    version = "~> 2.0.0"
```

```
Provider "aws" v3.0.0 is not compatible with Terraform v0.12.0.
Provider version v2.34.0 is the latest compatible version. Select 
it with the following constraint:

    version = "~> 2.34.0"

Alternatively, upgrade to the latest version of Terraform for compatibility with newer provider releases.
```

The above messages are for plugins installed via `terraform init` from a
Terraform registry, where the registry API allows Terraform Core to recognize
the protocol compatibility for each provider release. For plugins that are
installed manually to a local plugin directory, Terraform Core has no way to
suggest specific versions to upgrade or downgrade to, and so the error message
is more generic:

```
The installed version of provider "example" is not compatible with Terraform v0.12.0.

This provider was loaded from:
     /usr/local/bin/terraform-provider-example_v0.1.0
```

## Adding/removing major version support in SDK and Providers

The set of supported major versions is decided by the SDK used by the plugin.
Over time, SDKs will add support for new major versions and phase out support
for older major versions.

In doing so, the SDK developer passes those capabilities and constraints on to
any provider using their SDK, and that will in turn affect the compatibility
of the plugin in ways that affect its semver-based version numbering:

- If an SDK upgrade adds support for a new provider protocol, that will usually
  be considered a new feature and thus warrant a new minor version.
- If an SDK upgrade removes support for an old provider protocol, that is
  always a breaking change and thus requires a major release of the provider.

For this reason, SDK developers must be clear in their release notes about
the addition and removal of support for major versions.

Terraform Core also makes an assumption about major version support when
it produces actionable error messages for users about incompatibilities:
a particular protocol major version is supported for a single consecutive
range of provider releases, with no "gaps".

## Using the protobuf specifications in an SDK

If you wish to build an SDK for Terraform plugins, an early step will be to
copy one or more `.proto` files from this directory into your own repository
(depending on which protocol versions you intend to support) and use the
`protoc` protocol buffers compiler (with gRPC extensions) to generate suitable
RPC stubs and types for your target language.

For example, if you happen to be targeting Python, you might generate the
stubs using a command like this:

```
protoc --python_out=. --grpc_python_out=. tfplugin5.1.proto
```

You can find out more about the tool usage for each target language in
[the gRPC Quick Start guides](https://grpc.io/docs/quickstart/).

The protobuf specification for a version is immutable after it has been
included in at least one Terraform release. Any changes will be documented in
a new `.proto` file establishing a new protocol version.

The protocol buffer compiler will produce some sort of library object appropriate
for the target language, which depending on the language might be called a
module, or a package, or something else. We recommend to include the protocol
major version in your module or package name so that you can potentially
support multiple versions concurrently in future. For example, if you are
targeting major version 5 you might call your package or module `tfplugin5`.

To upgrade to a newer minor protocol version, copy the new `.proto` file
from this directory into the same location as your previous version, delete
the previous version, and then run the protocol buffers compiler again
against the new `.proto` file. Because minor releases are backward-compatible,
you can simply update your previous stubs in-place rather than creating a
new set alongside.

To support a new _major_ protocol version, create a new package or module
and copy the relevant `.proto` file into it, creating a separate set of stubs
that can in principle allow your SDK to support both major versions at the
same time. We recommend supporting both the previous and current major versions
together for a while across a major version upgrade so that users can avoid
having to upgrade both Terraform Core and all of their providers at the same
time, but you can delete the previous major version stubs once you remove
support for that version.

**Note:** Some of the `.proto` files contain statements about being updated
in-place for minor versions. This reflects an earlier version management
strategy which is no longer followed. The current process is to create a
new file in this directory for each new minor version and consider all
previously-tagged definitions as immutable. The outdated comments in those
files are retained in order to keep the promise of immutability, even though
it is now incorrect.

## Updating the plugin protocol in Terraform core

> Note: This section's audience is contributors to Terraform, not developers creating a new SDK.

New features added to Terraform core often require an update to the plugin protocol. These changes are reflected in a new minor version of the plugin protocol.

A new minor release of Terraform should only introduce one new minor release of the plugin-protocol. The minor release numbers of the plugin protocol and Terraform do not match, but they should have a one-to-one relationship.

### Add updates to a new plugin protocol minor version

1) Edit the `.proto` files for Protocol 5 and 6 in `docs/plugin-protocol/`.
    * If this is the first bigger change after a Terraform release, you may want to increase the minor protocol version in the file header
1) Commit your changes.
1) Run `make protobuf`.
    * This will use the symlinks in the `internal/tfplugin*` directories to access the latest minor versions' `.proto` files.
    * You should see diffs in `internal/tfplugin5/tfplugin5.pb.go` and `internal/tfplugin6/tfplugin6.pb.go`.
1) Run `make generate`.
    * You should see diffs in `internal/plugin/mock_proto/mock.go` and `internal/plugin6/mock_proto/mock.go`.