Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trouble shooting section for Percona Everest #157

Draft
wants to merge 101 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
0f766c5
Trouble shooting section fpr Everest
rasika-chivate Aug 20, 2024
cdf090f
Update troubleshoot.md
rasika-chivate Aug 20, 2024
b857443
Update troubleshoot.md
rasika-chivate Aug 20, 2024
315a285
installtion topic
rasika-chivate Aug 20, 2024
03cb752
Update troubleshoot.md
rasika-chivate Nov 8, 2024
3cacf6e
Update troubleshoot.md
rasika-chivate Nov 8, 2024
4200ddd
Update troubleshoot.md
rasika-chivate Nov 11, 2024
01c54b8
Update troubleshoot.md
rasika-chivate Nov 11, 2024
bcdf442
OLM subsription
rasika-chivate Nov 11, 2024
1c51f23
Update troubleshoot.md
rasika-chivate Nov 11, 2024
f951fa6
add a note
rasika-chivate Nov 11, 2024
e0d0212
Update mkdocs-base.yml
rasika-chivate Nov 11, 2024
a06fb6e
Update mkdocs-base.yml
rasika-chivate Nov 11, 2024
e45c83b
Update troubleshoot.md
rasika-chivate Nov 11, 2024
6cd0fb3
Update troubleshoot.md
rasika-chivate Nov 11, 2024
84e8520
Update troubleshoot.md
rasika-chivate Nov 11, 2024
021c88b
Update troubleshoot.md
rasika-chivate Nov 11, 2024
f3df0ed
Update troubleshoot.md
rasika-chivate Nov 11, 2024
2c163d9
formatting fix
rasika-chivate Nov 11, 2024
2ea730a
Update troubleshoot.md
rasika-chivate Nov 11, 2024
37ba5e8
Update troubleshoot.md
rasika-chivate Nov 11, 2024
49567db
Update troubleshoot.md
rasika-chivate Nov 11, 2024
7a67e28
Update troubleshoot.md
rasika-chivate Nov 11, 2024
b07110b
Update troubleshoot.md
rasika-chivate Nov 11, 2024
6ac2aae
Update troubleshoot.md
rasika-chivate Nov 11, 2024
22da411
Update troubleshoot.md
rasika-chivate Nov 11, 2024
9737cc1
Update troubleshoot.md
rasika-chivate Nov 11, 2024
b878fc2
Update troubleshoot.md
rasika-chivate Nov 11, 2024
20dcaef
Update troubleshoot.md
rasika-chivate Nov 11, 2024
ebaf2ee
Update troubleshoot.md
rasika-chivate Nov 11, 2024
9d17947
Update troubleshoot.md
rasika-chivate Nov 11, 2024
cec9785
Update troubleshoot.md
rasika-chivate Nov 11, 2024
16c99d6
Update troubleshoot.md
rasika-chivate Nov 11, 2024
44f0095
Update troubleshoot.md
rasika-chivate Nov 11, 2024
ed2a006
Update troubleshoot.md
rasika-chivate Nov 11, 2024
9cde83b
Update troubleshoot.md
rasika-chivate Nov 11, 2024
c8328bf
Update troubleshoot.md
rasika-chivate Nov 11, 2024
a10f977
added faq topic
rasika-chivate Nov 11, 2024
8026f7c
Update mkdocs-base.yml
rasika-chivate Nov 11, 2024
3ae21b2
added faq topic
rasika-chivate Nov 11, 2024
668d57a
Update faq.md
rasika-chivate Nov 11, 2024
c2771d2
Update troubleshoot.md
rasika-chivate Nov 18, 2024
e10b027
Update troubleshoot.md
rasika-chivate Nov 18, 2024
e01e395
Update faq.md
rasika-chivate Nov 22, 2024
01ad1b9
Update faq.md
rasika-chivate Nov 22, 2024
8522f77
Update faq.md
rasika-chivate Nov 22, 2024
13154e2
formatting fixes
rasika-chivate Nov 22, 2024
982e591
Update faq.md
rasika-chivate Nov 22, 2024
2326708
Update faq.md
rasika-chivate Nov 22, 2024
f0f620e
Update faq.md
rasika-chivate Nov 22, 2024
2f0cb72
Update faq.md
rasika-chivate Nov 22, 2024
7eb4654
Update faq.md
rasika-chivate Nov 22, 2024
4690af4
Update faq.md
rasika-chivate Nov 22, 2024
4052de9
Update faq.md
rasika-chivate Nov 22, 2024
a5e4d7b
Update faq.md
rasika-chivate Nov 22, 2024
64e3e06
Update faq.md
rasika-chivate Nov 22, 2024
fca4597
Update faq.md
rasika-chivate Nov 22, 2024
10b7285
formatting updates
rasika-chivate Nov 22, 2024
94bf12b
Update faq.md
rasika-chivate Nov 22, 2024
19f019f
Update faq.md
rasika-chivate Nov 22, 2024
e395913
Update faq.md
rasika-chivate Nov 22, 2024
bb0bb2e
Update faq.md
rasika-chivate Nov 22, 2024
bbd615f
Update faq.md
rasika-chivate Nov 22, 2024
becbe7c
Update faq.md
rasika-chivate Nov 22, 2024
178530e
Update faq.md
rasika-chivate Nov 22, 2024
46341ad
Update faq.md
rasika-chivate Nov 22, 2024
0930174
Update faq.md
rasika-chivate Nov 22, 2024
8b9a0e3
Update faq.md
rasika-chivate Nov 22, 2024
6c196a5
added troubleshoot topic to the home page
rasika-chivate Nov 22, 2024
d350d9b
Update index.md
rasika-chivate Nov 22, 2024
5038213
Update troubleshoot.md
rasika-chivate Nov 25, 2024
ecbbbbb
Update faq.md
rasika-chivate Nov 25, 2024
7174eb2
Update faq.md
rasika-chivate Nov 25, 2024
039c8f9
Update faq.md
rasika-chivate Nov 25, 2024
a9b3696
Update faq.md
rasika-chivate Nov 25, 2024
c23f07b
Update troubleshoot.md
rasika-chivate Nov 25, 2024
880bdc8
Update docs/reference/troubleshoot.md
rasika-chivate Nov 25, 2024
cace6e2
removed $ from all the coomands
rasika-chivate Nov 25, 2024
467b099
Merge branch 'trouble_shooting+everest' of https://github.com/percona…
rasika-chivate Nov 25, 2024
9de66f9
Update docs/reference/troubleshoot.md
rasika-chivate Nov 25, 2024
1dc9909
Update docs/reference/faq.md
rasika-chivate Nov 25, 2024
3b6aac3
Update docs/reference/faq.md
rasika-chivate Nov 25, 2024
d7efb01
Update faq.md
rasika-chivate Nov 25, 2024
93b15c4
Update faq.md
rasika-chivate Nov 25, 2024
b9b6102
Update faq.md
rasika-chivate Nov 26, 2024
e07241f
Update troubleshoot.md
rasika-chivate Nov 26, 2024
195a71a
Update faq.md
rasika-chivate Nov 26, 2024
908bdf5
Update faq.md
rasika-chivate Nov 26, 2024
ab50b67
moved troubleshoot section to an upper level after the use section
rasika-chivate Nov 26, 2024
6ec5e01
Update mkdocs-base.yml
rasika-chivate Nov 26, 2024
20dd687
Update mkdocs-base.yml
rasika-chivate Nov 26, 2024
8293a76
Update mkdocs-base.yml
rasika-chivate Nov 26, 2024
91a7117
Update faq.md
rasika-chivate Nov 26, 2024
edbb01c
Update troubleshoot.md
rasika-chivate Nov 26, 2024
edf531c
Update index.md
rasika-chivate Nov 26, 2024
5ae12c1
Update faq.md
rasika-chivate Nov 26, 2024
ad66848
Update faq.md
rasika-chivate Nov 26, 2024
7d6120f
Update troubleshoot.md
rasika-chivate Nov 26, 2024
ef27e89
Update troubleshoot.md
rasika-chivate Nov 26, 2024
0229040
Update mkdocs-base.yml
rasika-chivate Nov 26, 2024
2e289a4
Update troubleshoot.md
rasika-chivate Nov 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,11 @@ Explore how our security features are designed to ensure the security of your ho

</div><div data-banner markdown>

### :material-account-supervisor: Manage users { .title }
### :material-frequently-asked-questions: Troubleshooting { .title }

Learn how to manage user accounts in Percona Everest.
Dive into our troubleshooting section, which is designed to guide you through any issues you may encounter while using Percona Everest.

[Manage user accounts :material-arrow-right:](administer/manage_users.md){ .md-button .md-button--primary }
[Troubleshoot :material-arrow-right:](troubleshoot/troubleshoot.md){ .md-button .md-button--primary }

</div><div data-banner markdown>

Expand Down
61 changes: 61 additions & 0 deletions docs/troubleshoot/faq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Frequently asked questions (FAQ)

This section outlines the most frequently asked questions (FAQs) about Percona Everest.


## What is DatabaseEngine?

Provide an example of **Database Engine Operations**. Does it manage database operators, such as upgrades, or is it related to Percona Everest resources?

**Answer:**
The `DatabaseEngine` is a resource that contains compatible PG/PXC/MongoDB versions for the current Percona Everest version. This information can be retrieved and updated.

Refer to our [API documentation](https://percona-everest.readme.io/reference/getkubernetesclusterresources) for in-depth information.

### Do we have logs of the API calls made?

- Accessing these logs will enable us to confirm whether an API call was initiated for a specific user operation and to identify any errors that may have occurred during that process.

**Answer:**
Yes, the Percona Everest backend has logs. It runs as a `percona-everest` deployment within the `everest-system `namespace.

!!! Note
Percona Everest logs are crucial for troubleshooting issues where an operation is completed, but the corresponding Everest operator resource hasn't been created for some reason.

- Where are these logs stored?

**Answer:**
You can retrieve the logs from the pods associated with this deployment.


### Do we have any logs to troubleshoot issues between the front end and the Percona Everest API?

**Answer:**
You can view these logs in the console of your web browser.

### Do we need to troubleshoot any resources that are not part of the Percona Everest operator image?

- Is there a resource that receives API calls and interacts with individual resources that we should check for troubleshooting and logs when issues arise?

- Does the Percona Everest API communicate directly with the Percona Everest operator resources?

- Should we correlate the **Percona operator** issue with the corresponding Percona Everest resource and address the problem from that point?

**Answer:**
All communication with Everest resources begins with the API.

The API is responsible for updating Everest resources, while the Everest operator continues to create resources for the corresponding database operators. When debugging, start with the API, then proceed to the Everest operator, and finally examine the individual database operators.


### Does Percona Everest deploy PMM servers?

While `MonitorConfig` is present, does this resource interact directly with each `pmm-agent` container on the database pods, deploy new PMM Servers, and modify their configuration?

**Answer:**

Percona Everest doesn't deploy PMM. Instead, we configure PMM agents in each DB deployment to communicate with an existing PMM server.

When communicating with PMM, we generate an API key to facilitate data transmission. This API key allows us to configure monitoring endpoints for individual database operators, enabling them to send data to PMM.

!!! note
PMM is an external service we support and does not necessarily fit into our architecture.
173 changes: 173 additions & 0 deletions docs/troubleshoot/troubleshoot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# Troubleshooting common issues


This section is your go-to resource for tackling common issues and finding solutions. For additional troubleshooting tips and known issues, see the Percona Everest [Release Notes](../release-notes/release_notes_index.md) and [known limitations](../reference/known_limitations.md) section.


## Comprehensive installation overview

We leverage the [Operator Lifecycle Manager (OLM)](https://olm.operatorframework.io/) to manage the operators. OLM is deployed explicitly to the `everest-olm` namespace.

The main components are:

* olm-operator
* catalog-operator
* packageserver

### Percona Everest installation

When you install Percona Everest, the following components are installed:
{.power-number}


1. The `olm-operator`, `catalog-operator`, and `packageserver` gets installed in the `everest-olm` namespace:

Execute the following command, all three deployments should be present in the `everest-olm` namespace

```
kubectl get deploy -n everest-olm
NAME READY UP-TO-DATE AVAILABLE AGE
olm-operator 1/1 1 1 5m38s
catalog-operator 1/1 1 1 5m37s
packageserver 2/2 2 2 3m2s
```

2. The `everest-catalog` is installed, which serves as the repository for all the operators that Percona Everest supports.

Execute the following command, `everest-catalog` should be present in the `everest-olm` namespace:

```sh
kubectl get catalogsources -n everest-olm
NAME DISPLAY TYPE PUBLISHER AGE
everest-catalog Everest Catalog grpc Percona 6m5s
```

### OLM subscription

The process of installing an operator with OLM (Operator Lifecycle Manager) begins with creating a [subscription](https://olm.operatorframework.io/docs/concepts/crds/subscription/). OLM will reconcile this subscription and generate an installation plan. Once the installation plan is approved, OLM will create a Cluster Service Version (CSV), which installs everything necessary for the operator to function properly.


In addition to OLM, Percona Everest consists of these operators:

- Percona Operator for MySQL (PXC)
- Percona Operator for MongoDB (PSMDB)
- Percona Operator for PostgreSQL (PG)


In this section, we group all the database (DB) operators together because they are managed in the same way. When you run the command `everestctl install`, you must choose which operators to install (PXC, PSMDB, PG) and specify the namespaces for each. The following example assumes that you installed all the operators in the `everest` namespace:

??? example "Examples"
```
kubectl get sub -n everest
NAME PACKAGE SOURCE CHANNEL
percona-xtradb-cluster-operator percona-xtradb-cluster-operator everest-catalog stable-v1
percona-server-mongodb-operator percona-server-mongodb-operator everest-catalog stable-v1
percona-postgresql-operator percona-postgresql-operator everest-catalog stable-v2
```

```
kubectl get ip -n everest
NAME CSV APPROVAL
APPROVED
install-tjwjf percona-xtradb-cluster-operator.v1.14.0 Manual
true
install-8cl7m percona-server-mongodb-operator.v1.15.0 Manual
true
install-8lz2n percona-postgresql-operator.v2.3.1. Manual
true
```

```
kubectl get csv -n everest
NAME DISPLAY
VERSION REPLACES PHASE
percona-xtradb-cluster-operator.v1.14.0 Percona Operator for MySQL based on Percona XtraDB Cluster
1.14.0 percona-xtradb-cluster-operator.v1.13.0 Succeeded
percona-server-mongodb-operator.v1.15.0 Percona Distribution for MongoDB Operator
1.15.0 percona-server-mongodb-operator.v1.14.0 Succeeded
percona-postgresql-operator.v2.3.1 Percona Operator for PostgreSQL
2.3.1 Succeeded
everest-operator.v0.10.1 Everest operator
0.10.1 everest-operator.v0.9.0 Succeeded
```

```
kubectl get deploy -n everest
NAME READY UP-TO-DATE AVAILABLE AGE
percona-xtradb-cluster-operator 1/1 1 1 23m
percona-server-mongodb-operator 1/1 1 1 22m
percona-postgresql-operator 1/1 1 1 21m
```

## Database deployment

In Everest, every database begins with a DatabaseCluster (DBC) Custom Resource (CR) that is deployed to the chosen namespace. Depending on the selected engine type, this DBC will be converted into a corresponding Custom Resource that can be understood by the database operator, whether it be `PXC`, `PSMDB`, or `PG`.

Example:
This gives a step by step approach to troubleshoot if something goes wrong with your database deployment:
{.power-number}

1. Run the following command:

```
kubectl get db -n everest
NAME SIZE READY STATUS HOSTNAME AGE
mysql-7tl 2 2 ready mysql-7tl-haproxy.everest 6m8s

kubectl get pxc -n everest
NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE
mysql-7tl mysql-7tl-haproxy.everest ready 1 1 6m10s
```

2. If an issue arises, first investigate these two objects by executing the following commands:

```
kubectl describe db/mysql-7tl -n everest
kubectl describe pxc/mysql-7tl -n everest
```

3. If relevant information is not found here, then check the Everest Operator logs as well as the PXC Operator logs:

```
kubectl logs pods/everest-operator-controller-manager-5b868c4fcc-rt6rp -n everest-system
kubectl logs pods/percona-xtradb-cluster-operator-77c5ffddf6-fv8vg -n everest
```

!!! note

The names of the pods in your deployment may vary. If nothing relevant appears, check the logs for details.


## Backups and restores

### Backups

Every backup in percona Everest begins with the creation of a `DatabaseClusterBackup (DBB)` `Custom Resource (CR)` that is deployed to a selected namespace. Depending on the chosen engine type, this `DBB` will be converted into the corresponding Custom Resource that can be interpreted by the database operator, such as `pxc-backup`, `psmdb-backup`, or `pg-backup`.

```
kubectl describe pxc-backup
kubectl describe psmdb-backup
kubectl describe pg-backup
```

### Restores

Every Restore operation in Everest begins with a `DatabaseClusterRestore (DBR)` Custom Resource (CR) that is deployed in the specified namespace. Based on the selected engine type, this `DBR` will be transformed into the corresponding Custom Resource that can be processed by the database operator, such as `pxc-restore`, `psmdb-restore`, or `pg-restore`.

```sh
kubectl describe pxc-restore restoreName
kubectl describe psmdb-restore restoreName
kubectl describe pg-restore restoreName
```












6 changes: 6 additions & 0 deletions mkdocs-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,11 @@ nav:
- Monitoring: use/monitor_endpoints.md
- Multiple namepaces: use/multi-namespaces.md
- API rate limiting: use/API_rate_limit.md

- Resolve issues:
- Troubleshooting: troubleshoot/troubleshoot.md
- FAQ: troubleshoot/faq.md

- API: API.md
- Reference:
- Limitations: reference/known_limitations.md
Expand All @@ -202,5 +207,6 @@ nav:
- Copyright and licensing information: reference/copyright.md
- Trademark and copyright policy: reference/trademark-policy.md


- Contribute: contribute.md