-
Notifications
You must be signed in to change notification settings - Fork 34
/
Copy path_mysql-monitor-health-pas.html.md.erb
126 lines (83 loc) · 5.71 KB
/
_mysql-monitor-health-pas.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
## <a id="node-health"></a> Node Health
When determining where to route traffic, the proxy queries an HTTP healthcheck process running on the node. This healthcheck can return as either healthy or unhealthy, or the node can be unresponsive.
### <a id="healthy"></a> Healthy
If the healthcheck process returns HTTP status code `200`, the proxy includes the node in its pool of healthy nodes.
When a new or resurrected nodes rejoin the cluster, the proxy continues to route all connections to the currently active node. In the case of failover, the proxy considers all healthy nodes as candidates for new connections.
<%= image_tag "images/switchboard-all-healthy.png" %>
### <a id="unhealthy"></a> Unhealthy
If the healthcheck returns HTTP status code `503`, the proxy considers the node unhealthy.
<!-- We will eventually move the following link to a troubleshooting topic in 2.5 -->
This happens when a node becomes non-primary. For more information, see [Cluster Scaling Behavior](https://docs.pivotal.io/p-mysql/1-10/architecture.html#scale-behavior).
The proxy severs existing connections to newly unhealthy node. The proxy routes new connections to a healthy node, assuming such a node exists. Clients are expected to handle reconnecting on connection failure should the entire cluster become inaccessible.
<%= image_tag "images/switchboard-unhealthy.png" %>
### <a id="unresponsive"></a> Unresponsive
If node health cannot be determined due to an unreachable or unresponsive healthcheck endpoint, the proxy considers the node unhealthy. This may happen if there is a network partition or if the VM running the node and healthcheck died.
## <a id="monitor"></a> Monitor Node Health
You can monitor the health status of your nodes by doing one of the following:
+ [Monitor Node Health Using the Dashboard](#dashboard)
+ [Monitor Node Health Using the API](#proxy-api)
### <a id='dashboard'></a> Monitor Node Health Using the Dashboard
#### <a id='dashboard-url'></a> Construct the URL for the Dashboard
Each MySQL Proxy instance has its own Switchboard dashboard. To access the dashboard, you must construct the dashboard URL.
To construct the URL for the Switchboard dashboard, do the following:
1. Find and record the system domain on the **Domains** pane in the PAS tile.
1. Find and record the number of MySQL Proxy VM instances in your deployment in the **Resource Config** pane.
1. Use the above information to build the Switchboard dashboard URL as follows: `https://[PROXY-VM-INDEX]-proxy-p-mysql-ert.[SYSTEM-DOMAIN]`
#### <a id='dashboard-login'></a> Log In to the Dashboard
To log into the Switchboard dashboard, do the following:
1. In the PAS tile, the URL in your browser contains the Ops Manager fully qualified domain name (FQDN) and the PAS product GUID, as in the following example: `https://OPS-MANAGER-FQDN/products/PAS-PRODUCT-GUID/az_and_network_assignments/edit`<br>
To log into the dashboard, you need the username and password for the MySQL Proxy VM that are stored in the BOSH Director CredHub.<br>
To obtain them, enter `https://[OPS-MANAGER-FQDN]/api/v0/deployed/products/[PAS-PRODUCT-GUID]/variables?name=mysql-proxy-dashboard-credentials` into your browser and record the username and password.
1. Enter the dashboard URL you constructed into your browser and log in using the credentials you obtained in [Construct the URL for the Dashboard](#dashboard-url).
#### <a id="use-dashboard"></a> Use the Dashboard
To monitor node health using the Switchboard dashboard, do the following:
1. To view a list of proxies in your browser, navigate to the URL that you created in [Construct the URL for the Dashboard](#dashboard-url).
1. When prompted, enter the username and password that you recorded in [Log In to the Dashboard](#dashboard-login).
1. Click the link for the proxy that you want to use to monitor node health.
1. If you are prompted, enter the username and password that you recorded in [Log In to the Dashboard](#dashboard-login).
<%= image_tag "images/switchboard-all-healthy.png" %>
### <a id="proxy-api"></a> Monitor Node Health Using the API
You can also use the Switchboard API to obtain the information that is shown on the Switchboard dashboard. For example, you might want to use the API to write your own app to monitor the cluster.
To monitor node health using the Switchboard API, do the following:
1. To monitor node health, run the following command:
```
curl https://USERNAME:PASSWORD@N-HOSTNAME/v0/backends
```
Where:
+ `USERNAME` is the `username` you recorded in [Log In to the Dashboard](#dashboard-login).
+ `PASSWORD` is the `password` you recorded in [Log In to the Dashboard](#dashboard-login).
+ `N` is either 0, 1, or 2 depending on the proxy you want to connect to.
+ `HOSTNAME` is the IP address of the **Database** found in the **Status** tab of your PAS tile.
The above command outputs a JSON object similar to the following:
<pre class="terminal">
$ curl https://abcdefghijklmno:012345678912345@0-proxy.123abc45-67d8-912e-34f5-g34612c10dba.org.dedicated-mysql.cf-app.com/v0/backends
[
{
"host": "a-b1234c5d6.e-f891.bosh",
"port": 6033,
"healthy": true,
"name": "backend-0",
"currentSessionCount": 0,
"active": true,
"trafficEnabled": true
},
{
"host": "a-b1234c5d6.e-f891.bosh",
"port": 6033,
"healthy": true,
"name": "backend-1",
"currentSessionCount": 0,
"active": false,
"trafficEnabled": true
},
{
"host": "a-b1234c5d6.e-f891.bosh",
"port": 6033,
"healthy": true,
"name": "backend-2",
"currentSessionCount": 0,
"active": false,
"trafficEnabled": true
}
]
</pre>