8/23/2020 Migrating to v1 metadata server endpoint Migrating to v1 metadata server endpoint ng: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on Septemb Migration overview To migrate to v1.0, complete the following steps: 1. Identify the VM instances (#nd-vm) that are using the deprecated metadata server endpoints. If you have Google Kubernetes Engine clusters (/kubernetes-engine/docs/concepts/kubernetes-engine-overview), you need to identify the GKE nodes (#nd-gke-node) that are using the deprecated endpoint. 2. On the identied VM instances or nodes, nd the processes (#nd-process), applications (#apps-to-update), or images (#image-update) that are using the deprecated metadata server endpoints. For processes and applications, review the following information: If the process belongs to an application that you did not develop, update the application to use a v1 metadata server endpoint. For a list of known applications that require an update, see applications that require an update (#apps-to-update). If the process belongs to an application that you developed and your code is calling any of the legacy endpoints, complete the following steps: a. Review the differences between v1 and the deprecated metadata server endpoints. For v1beta1, see Differences between v1beta1 and v1.0 (#diff-v1beta1). For v0.1, see Differences between v0.1 and v1.0 (#diff-v0.1). b. Update queries to use the v1 metadata server endpoint. https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server/ 1/19 8/23/2020 Migrating to v1 metadata server endpoint If the process belongs to an application that you developed but your code is not making requests to any of the legacy endpoints, then the requests might be made by SDKs or other dependencies. To x this, update all SDKs and dependencies used by the application.To view the versions of the Google libraries that require an update, see supported library versions (#supported-google-libraries). 3. (Optional) Disable the v0.1 and v1beta1 (/compute/docs/storing-retrieving-metadata#disable-legacy-endpoints) metadata server endpoints. Identifying the VM instances To identify which VM instances are using v0.1 and v1beta1 endpoints, you can make requests to two new endpoints. The output from querying these new endpoints tells you how many times a given VM instance accessed the deprecated endpoints. Note: Each time a VM instance is stopped, the counter resets. To see if a VM instance has accessed the v0.1 and v1beta1 endpoints, run the following commands: curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetada curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetada The following python sample shows how to programmatically watch these new endpoints: compute/metadata/detect_legacy_usage.py (https://github.com/GoogleCloudPlatform/python-docs- samples/blob/master/compute/metadata/detect_legacy_usage.py) om/GoogleCloudPlatform/python-docs-samples/blob/master/compute/metadata/detect_legacy_usage.py) def wait_for_legacy_usage(callback): url = '{}/instance/legacy-endpoint-access'.format(METADATA_URL) last_etag = '0' https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server/ 2/19 8/23/2020 Migrating to v1 metadata server endpoint counts = None while True: r = requests.get( url, params={ 'last_etag': last_etag, 'recursive': True, 'wait_for_change': True }, headers=METADATA_HEADERS) if r.status_code == 503: # Metadata server unavailable print('Metadata server unavailable. Sleeping for 1 second.') time.sleep(1) continue if r.status_code == 404: # Feature not yet supported print('Legacy endpoint access not supported. Sleeping for 1 hour.') time.sleep(3600) continue r.raise_for_status() last_etag = r.headers['etag'] access_info = json.loads(r.text) if not counts: counts = access_info if access_info != counts: diff = { ver: access_info[ver] - counts[ver] for ver in counts } counts = access_info callback(diff) Identifying the GKE nodes To identify the nodes in your Google Kubernetes Engine clusters that are using the v0.1 and v1beta1 endpoints, see Identifying workloads using the legacy metadata server endpoints (/kubernetes-engine/docs/how-to/protecting-cluster-metadata#identify-legacy-apis). Identifying the processes https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server/ 3/19 8/23/2020 Migrating to v1 metadata server endpoint After you have identied the VM instances that are making requests to the deprecated metadata server endpoints, you can then nd the processes on these VM instances that are making these requests. To identify the processes, use logging or monitoring tools such as ngrep or auditd. ngrepauditd (Linux only) (#auditd-li… You can use ngrep (network grep) to collect packets and lter the contents of these packets. To collect packet information for processes that make requests to the deprecated endpoints, on your VM instance, run the following command: sudo ngrep -l -q "v1beta1|0\.1/meta-data" tcp \ and dst host 169.254.169.254 or metadata.google.internal \ and dst port 80 If any requests are made to the deprecated endpoints, the output resembles the following: T 10.128.0.4:41312 -> 169.254.169.254:80 [AP] GET /computeMetadata/v1beta1/instance/id HTTP/1.1..Host: metadata.goog..User-Ag The output contains the following information: The root url: /computeMetadata/v1beta1/instance/id. The user agent that is used to make the request. In this example, it is curl/7.52.1. The IP address (source IP) and port for the VM instance that the request originated from. In this example, the IP address is 10.128.0.4 and the port is 41312 . If the process is still alive, you can review the port information to nd the process ID. Reviewing port information The following procedure might identify processes with outstanding hanging HTTP GET (/compute/docs/storing-retrieving-metadata#waitforchange) requests. This procedure nds the processes that are talking to both the current and deprecated metadata endpoints. To list the sockets with open connections to the metadata server, on your VM instance, run the following command: sudo lsof -n -P +c 0 -i @169.254.169.254 https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server/ 4/19 8/23/2020 Migrating to v1 metadata server endpoint The output resembles the following: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME google_network_ 798 root 7u IPv4 1674967626 0t0 TCP 10.128.0.4:44876- google_accounts 805 root 5u IPv4 1674980506 0t0 TCP 10.128.0.4:44878- google_clock_sk 809 root 5u IPv4 1674982496 0t0 TCP 10.128.0.4:44880- google_clock_sk 809 root 6u IPv4 1674914460 0t0 TCP 10.128.0.4:44874- The NAME column shows the source IP, destination IP, and ports. If this isn't enough to identify a process, you can use the ps command to view more information. ps 798 805 809 The output resembles the following: PID TTY STAT TIME COMMAND 798 ? Ss 9:07 /usr/bin/python /usr/bin/google_network_daemon 805 ? Ss 14:19 /usr/bin/python /usr/bin/google_accounts_daemon 809 ? Ss 4:33 /usr/bin/python /usr/bin/google_clock_skew_daemon Dierences between the v1beta1 and v1.0 metadata server endpoints The v1 metadata server functions slightly differently than the previous v1beta1 server. The new metadata server requires that all requests provide the Metadata-Flavor: Google header, which indicates that the request was made with the intention of retrieving metadata values. Update your requests to include this new header. For example, a request to the disks/ attribute now looks like the following: user@myinst:~$ curl "http://metadata.google.internal/computeMetadata/v1/instanc https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server/ 5/19 8/23/2020 Migrating to v1 metadata server endpoint Dierences between the v0.1 and v1.0 metadata server endpoints In general, the v0.1 metadata server endpoints differ from v1 in the following ways: The root used for querying metadata server endpoint has changed from http://metadata.google.internal/0.1/meta-data/ to http://metadata.google.internal/computeMetadata/v1/. v1 metadata server requests must include the Metadata-Flavor: Google header. For example, a request to the disks/ attribute now looks like the following: curl "http://metadata.google.internal/computeMetadata/v1/instance/disks/" Custom metadata (/compute/docs/storing-retrieving-metadata#custom) can be queried at both the project and instance level (/compute/docs/storing-retrieving-metadata#project-instance-metadata). In the v0.1 metadata server, for the attributes/ property, both instance and project metadata values are in the same directory. If an attribute has both instance and project metadata values, the instance metadata value for a given key is returned. For v1 queries, the URL must specify whether instance or project metadata is requested. For example, to query the sshKeys attribute, run one or both of the following: For sshKeys that are set on the project, run the following command: curl "http://metadata.google.internal/computeMetadata/v1/project/attribute For sshKeys that are set on this instance, run the following command: curl "http://metadata.google.internal/computeMetadata/v1/instance/attribut Some metadata entries in v0.1 return JSON objects. In the v1 metadata server, these metadata entries are organized by directories. This organization of entries lets you https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server/ 6/19 8/23/2020 Migrating to v1 metadata server endpoint query specic index entries and values for these previously nested elds. For more information