- Published on
NSX-T Troubleshooting
- Authors
- Name
- Jackson Chen
How to replace NSX-T Local Manager Certificate with Self-Signed
https://topvcf.com/2022/09/01/replace-nsx-t-localmanager-certificate-with-a-self-signed/
# Preparation
1. Create snapshot for the NSX-T Managers
2. Do a NSX-T backup
a. Login to NSX-T manager
b. Navigate to
i. System -> Lifecycle Managment
ii. Select "Backup & Restore"
iii. Click "Start Backup"
#***** Process
# Create CSR
1. Login to NSX-T local manager as "admin"
https://<nsx-t-ip>/login?local=true
2. Navigate to
System -> Settings -> Certificates
3. Verify the LocalManager certificate for expiration day
4. Click CSRs tab, click Generate CSR
5. Enter the information
CN local-manager
Name LocalManager # This is the display name
Then enter other relevant information for your organization
6. click Save
# Self sign the CSR
1. From CSRs tab, select the newly created LocalManager CSR by click the "..."
2. Select Self Sign Certificate for CSR
3. Unselect/untick Service Certificate, and click Save
Important:
Ensure untick service certificate, otherwise will fail to install the LocalManager certificate
This is NOT a service certiticate
Service Certificate, such as load balancer, etc
# Copy the newly LocalManager certificate id
1. From Certificates tab, select the newly signed LocalManager certificate
2. Click to expand, and copy the certificate id
# Update NSX Local Manager to use the newly signed certificate
1. Open postman
2. Change API to POST
3. Enter the following in the text field
https://<nsx-mgr>/api/v1/trust-management/certificates?action=set_pi_certificate_for_federation
4. Click Authorization tab, change to basic auth
5. Enter nsx-t manager "admin" to Username, and enter the credential
6. Click Headers tab -> Change KEY to Content-Type -> Change VALUE application/json
7. click Body tab -> Change to "raw" -> Change to JSON
8. Copy the following to the body # change id to the new LocalManager certificate id
{ "cert_id": "<id>",
"service_type": "LOCAL_MANAGER" }
Important: The double quote is " # The straight quote
9. Click Send
Note: Should see 200 successful
# Verify the new LocalManager has been assigned
1. Go back to NSX-T Manager certificates, and do a refresh
2. You will notice the new LocalManager has "1" under "Where Used" column
Note:
If it is no used, then there is "0" under "Where Used" column
3. Verify the new LocalManager is in used/assigned, then
4. Delete the old certificate once confirmed it is not used anymore from nsx-t manager
Troubleshooting - Verify all the NSX-T manager certificates
# Get all certificates if used or not
GET https://<nsx-manager-IP>/api/v1/trust-management/certificates
# Verify just one certificate
GET https://<nsx-manager-IP>/api/v1/trust-management/certificates/<cert-id>
Example
GET https://172.168.1.236/api/v1/trust-management/certificates/dee8d78b-5e04-4deb-8d36-6b86f79f058b
# Verify ceriticate is valid
GET https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=validate
How to replace NSX-T 3.x with CA Signed Certificates
https://www.vexpert.cloud/how-to-replace-ca-signed-certificate-in-nsx-t-manager/
This shows the procedure to replace the self-sgined certificates for NSX-T 3.x manager cluster with Microsoft CA signed certificates.
# Pre-requisities
1. Create the certificate template in MS Certificat Authority server
2. Prepare the certificate signing request (CSR) file and issue the certificate based on the NSX-T template
# Import the CA signed certificate to NSX-T Manager
1. Login to NSX-Manager as "admin" and navigate to
System -> Settings -> Certificates
2. Upload the root certificate and intermediate certificate
i. Click Import -> Import CA Certificate
ii. Enter name for root certificate
iii. Copy the content of the root certificate ".cer" file in to the Certificate Contents textbox
iv. Toggle off "Service Certificate", and click Import
Important: Need to toggle off Service Certificate
Otherwise, it will fail to apply the NSX-T certificates
v. Import the Issuing CA certificate
Following the above steps to import the issuing CA certificate
3. Import three NSX-T manager certificate or a certificate has all required SAN
i. Click Import -> Import Certificate
ii. Enter a name for the CA signed NSX-T certificate
iii. In the Certificate Contents textbox, you copy the full chain of NSX-T manager certificate in the below order
---BEGIN CERTIFICATE--
NSX-T Manager or Cluster VIP cert
--END CERTIFICATE--
---BEGIN CERTIFICATE--
CA Issuing CA certificate
--END CERTIFICATE--
---BEGIN CERTIFICATE--
CA Root CA certificate
--END CERTIFICATE--
iv. In the Private Key text box, copy the contents of the <nsxmanager>.key file
---BEGIN PRIVATE KEY---
content of private key
---END PRIVATE KEY---
v. Toggle off Service Certificate, and click Import
4. Repeat the same steps to import the certificates for nsxmgr02, nsxmgr03, and nsxmgr (Cluster VIP FQDN)
5. Activate the imported CA-sgined certificates via REST API
After import all the certificates, we need to activate them via REST API calls.
i. Open Postman
ii. POST https://<nsx-mgr-FQDN>/api/v1/node/services/http?action=apply_certificate&certificate_id=<Certificate_id>
Note
Obtain the certificate id from NSX-T Manager -> Settings -> Certificates, and select the certificate
iii. Click Send
You will get 200 OK # if successful
iv. Repeat the same steps to activate the certificate for the remaining two NSX-T manager nodes.
vi. Active the NSX-T manager cluster VIP FQDN
Note: It is different request URL
POST https://<nsx_manager_FQDN>/api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=<cluster_certificate_id>
You will get 200 OK # if successful
6. Login to NSX-T Manager and NSX-T VIP FQDN and ensure the certificates have been updated.
NSX-T REST API Tips
https://spillthensx.com/nsx-t-rest-api-tips/
https://www.sysadmintutorials.com/nsx-t-api-host-to-use-postman-for-beginners/
https://developer.vmware.com/apis/1583/nsx-t
https://code.vmware.com/apis/892/nsx-t
- NSX-T Manager hosts its own REST API reference You will probably find launching the NSX-T Data Center API Guide from the NSX-T Manager Web UI more efficient than using the online version at: https://code.vmware.com/apis/892/nsx-t
# How to access NSX-T REST API reference
1. Login to NSX-T Manager
2. On the top-right, click "?" icon, and select "API Documentation" from the drop down list
3. It will open NSX-T Data Center API Guide
# How to search API reference
1. Navigate to the bottom left on the NSX-T Data Center API Guide
2. Click "Single paged API guide", and search
- Use the Postman App as your REST API Client Postman is a API application. It can be installed without creating an account
# Configure Postman
1. Open Postman, and click "Skip signing in and take me straight to the app"
2. Disable SSL certificate verification
2. Select Settings -> General
SSL certificat verification Set to "OFF"
3. Start by Create a request
4. Select Authorization tab, and select "Basic Auth" under Type, then
enter NSX-T "admin" user credential for Username and Password
5. Click Headers tab
i. Under KEY, select "Content Type"
ii. Under VALUE, select "application/json"
Note
This will ensure the requests will use JSON encoding,
the format expected by the NSX-T Data Center API
- Start with a GET to validate your setup As a functional test we are going to retrieve or GET the NSX-T Manager version:
i. using https://<nsx-manager-fqdn>/api/v1/node
ii. or https://<nsx-manager-ip-address>/api/v1/node
The goal is get a 200 OK Status response, along with a body that includes “product_version”.
We now have Postman correctly configured as an NSX-T REST API Client.
NSX-T REST API Tips - Intermediate Tips - HTTP Response messages and making changes
- HTTP Response Messages https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
All HTTP response status codes are separated into five classes or categories. The first digit of the status code defines the class of response, while the last two digits do not have any classifying or categorization role. There are five classes defined by the standard:
1xx informational response – the request was received, continuing process
2xx successful – the request was successfully received, understood, and accepted
3xx redirection – further action needs to be taken in order to complete the request
4xx client error – the request contains bad syntax or cannot be fulfilled
5xx server error – the server failed to fulfil an apparently valid request
PUSH or PUT In me methods, like the creation of a segment, we can use a PATCH or PUT command. The PUT command has an additional requirement, where the “_revision” property must be specified. In order to prevent one client from overwriting another client’s updates, NSX-T employs a technique called optimistic concurrency control.
Using Body in a PATCH or PUT To keep it simple, let’s use PATCH to create a segment named web-tier, using: PATCH /policy/api/v1/infra/segments/
For my lab, I’ve specified the PATCH as
https://nsxtmgr.core.hypervizor.com/policy/api/v1/infra/segments/web-tier
Select Body, raw, and JSON
Enter the Segments configuration details into the body of the message
Select Send to initiate the request
We are looking for a 200 OK response, to indicate that the Segment was created successfully
- Deleting Protected Objects using X-Allow-Overwrite Objects have a property named _create_user. This is typically nsx_policy, the NSX-T built-in Principal Identity. In this case, notice that this IP Block, named Finance-block, was created by a Principal Identity
From the respond body, there is "_create_user" property.
Attempting to delete the value of "_create_user" owned object results in 400 Bad Request since it is protected, with the error message”: “Principal ‘admin’ with role ‘[enterprise_admin]’ attempts to delete or modify an object of type IpAddressBlock it doesn’t own. (createUser=tenant1, allowOverwrite=null)”
# The solution, to delete the protected object,
add a new Header
KEY: X-Allow-Overwrite, with a VALUE of true
Result:
The 200 OK confirms the object has been deleted.
More on Principal Identities here: https://spillthensxt.com/introduction-to-nsx-t-principal-identities/
- Using Override and Cascade, in Unable to delete Logical Switch Typically it’s not possible to delete an object if there are dependencies on that object. For example, you can’t delete a logical switch if there are Guest VMs connected to that logical switch. In this example I’ve attempted a delete, resulting in the error: “Unable to delete Logical Switch LogicalSwitch/aa8c3d52-cacd-4392-a986-d774bf23089d because there are 1 logical ports on this switch. Please use the object name or UUID in Global Search to find all linked objects.”
# The solution is at add parameters to the delete:
?detach=true&cascade=true
detach = true
makes the delete forcible,
cascade = true
to delete all the related ports. You can see these in Postman as Params:
NSX-T REST API Tips - Advanced Tips
- Using the Management Plane API for determining the realized state of Policy API changes
This Policy API GET identifies that DHCP_Server is attached to Tier-1 Gateway lab-tier-1
GET https://nsxtmgr.lab/policy/api/v1/infra/tier-1s/lab-tier-1/
- Identifying the REST API method using Chrome Developer tools Launch Chrome Developer tools from Chrome as follows
1. Open Google Chrome
2. On the top right, select settings -> More Tools
3. On the select window, select "Developer Tools"
i. Select the Network tab and perform the operation using the NSX-T Web UI, in this case creating a segment.
2. Change the view to “Use small request rows”.
Notice that the segment named test-seg was created with the Policy API method /policy/api/v1/infra/segments