Skip to content

Commit

Permalink
Merge pull request #41 from appoptics/NH-3272-remove-appoptics-sdk
Browse files Browse the repository at this point in the history
NH-3272: Clean up Appoptics SDK
  • Loading branch information
jiwen624 authored May 9, 2022
2 parents 389cb1a + 17183ad commit 3b25110
Show file tree
Hide file tree
Showing 42 changed files with 58 additions and 2,523 deletions.
92 changes: 12 additions & 80 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,10 @@ This repository contains Solarwinds APM implementation that works with OpenTelem

Here is the summary of the sub-projects:
- agent : Builds the full OT auto agent with extra Solarwinds APM components. This is simply a repackaging build script that pull OT agent and our sub-projects to construct a new auto agent
- appoptics-opentelemetry-sdk : (Archived as we have decided not to maintain backward-compatibility with AppOptics) Builds the SDK artifact which has exactly the same interface as our existing AppOptics java agent SDK https://github.com/librato/joboe/tree/master/api . This is to be used as a base for `appoptics-opentelemetry-sdk-shaded`
- appoptics-opentelemetry-sdk-shaded : (Archived as we have decided not to maintain backward-compatibility with AppOptics) Same as appoptics-opentelemetry-sdk but can be used with the OT agent with shaded class names
- appoptics-opentelemetry-sdk : (most of which are archived as we have decided not to maintain backward-compatibility with AppOptics) Current it has only one API: the agent checker.
- custom : Extra Solarwinds APM components, contains all custom functionality, SPI and other extensions (for example Sampler, Tracer Provider etc) to be loaded by OT's agent classloader
- core-bootstrap : Core Solarwinds APM components that need to be made available to bootstrap classloader. This is important for `appoptics-opentelemetry-sdk` as the classes from `appoptics-opentelemetry-sdk` are loaded by app loader, which has no access to OT's agent classloader which loads `custom`
- instrumentation : Additional instrumentation provided by us using the OT instrumentation framework (ByteBuddy)
- sdk-extensions : Builds the AO extension jar which runs with the original OT agent (vs the agent built from `agent` sub-project)
- sdk-extensions-bootstrap : Builds the AO extension jar dependencies that should be made available to bootstrap loader. It basically package the `core` and `metrics` from joboe

More details for each of the sub-projects are listed in [Sub-Projects](#sub-projects) section

Expand All @@ -19,68 +16,31 @@ More details for each of the sub-projects are listed in [Sub-Projects](#sub-proj
#### Preparations
Since this project has dependencies on various internal artifacts from [joboe](https://github.com/librato/joboe), the build machine would need access to those artifacts. Currently the Joboe core libraries for the OpenTelemetry custom distro are in the `otel` branch of the `Joboe` repo and are published to the Github Packages.

#### Agent/Extensions Jars
Simply run `gradle build` at the root folder.
#### Agent Jars
Simply run `gradle build` at the root folder. (or run `gradle publishToMavenLocal` to publish to local maven repo)

The agent should be built at `agent\build\libs\solarwinds-apm-agent-all.jar`.
The sdk-extensions jar should be built at `sdk-extensions\build\libs\sdk-extensions-1.0-SNAPSHOT-all.jar`.

#### SDK artifacts
To build the `appoptics-opentelemetry-sdk` and `appoptics-opentelemetry-sdk-shaded` artifacts, use
`gradle :appoptics-opentelemetry-sdk:publishToMavenLocal` and
`gradle :appoptics-opentelemetry-sdk-shaded:publishToMavenLocal`

The artifacts will be published to local maven repo. (Use the Github Actions workflow `Release` to build and publish the custom distro to Github Packages.)
The agent should be built at `agent\build\libs\solarwinds-apm-agent.jar`.

## Usage
#### Agent Jar
Attach the agent to jvm process arg such as:
`-javaagent:"_the_path_to_the_jar_file" -Dotel.solarwinds.service.key=<service key here>`

Upon successful initialization, the log should print such as:
```
[otel.javaagent 2021-06-30 13:04:07:759 -0700] [main] INFO com.appoptics.opentelemetry.extensions.AppOpticsTracerProviderConfigurer - Successfully initialized Solarwinds APM OpenTelemetry extensions with service key ec3d********************************************************5468:ot
```
#### Extension Jar
1. Either download the auto agent directly from https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent-all.jar or build from source. To build from source:
- Checkout https://github.com/open-telemetry/opentelemetry-java-instrumentation
Navigate to the project directory, then gradlew build -x test (-x test is for skipping tests, remove the option if tests are to be run). Use java 9 or above => export JAVA_HOME=<Java 9+ path>.
- The java agent jar can be found in ...\git\opentelemetry-java-instrumentation\javaagent\build\libs
2. Attach the OT agent and our AO extension to the jvm process arg such as:
```
-javaagent:"C:\Users\patson.luk\Downloads\opentelemetry-javaagent-all-1.3.1.jar"
-Dotel.javaagent.experimental.initializer.jar="C:\Users\patson.luk\git\opentelemetry-custom-distro\sdk-extensions\build\libs\sdk-extensions-1.0-SNAPSHOT-all.jar"
-Dotel.traces.exporter=appoptics
-Dotel.traces.sampler=appoptics
-Dotel.metrics.exporter=none
-Dotel.propagators=tracecontext,baggage,appoptics
-Dotel.appoptics.service.key=<service key here>
-Xbootclasspath/a:"C:\Users\patson.luk\git\opentelemetry-custom-distro\sdk-extensions-bootstrap\build\libs\sdk-extensions-bootstrap-1.0-SNAPSHOT.jar"
```
The service key can also be defined via the environment variable `SW_APM_SERVICE_KEY` or in the config file.

Upon successful initialization, the log should print such as:
```
[otel.javaagent 2021-07-08 10:41:16:650 -0700] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.3.1
[otel.javaagent 2021-07-08 10:41:19:141 -0700] [main] INFO com.appoptics.opentelemetry.extensions.AppOpticsTracerProviderConfigurer - Successfully initialized AppOptics OpenTelemetry extensions with service key ec3d********************************************************5468:ot
[otel.javaagent 2021-06-30 13:04:07:759 -0700] [main] INFO com.appoptics.opentelemetry.extensions.AppOpticsTracerProviderConfigurer - Successfully initialized Solarwinds APM OpenTelemetry extensions with service key ec3d********************************************************5468:ot
```

**There are certain challenges for classloading with extension jar, more details in https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/3350**
#### SDK
The custom distro supports the standard OpenTelemetry APIs/SDKs. In addition to that, it also has an API to check if the agent is ready to use (and wait for a specified duration before it is ready). This is useful when you need to create traces manually.

#### SDK artifact
After the artifact `appoptics-opentelemetry-sdk-shaded` is built and published to local maven, it can be used by any java maven projects by adding `dependency` to `pom.xml` such as:
```
<dependency>
<groupId>com.appoptics.agent.java</groupId>
<artifactId>appoptics-opentelemetry-sdk-shaded</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
```
The API can be called as below, which waits for at most 10 seconds before the agent is ready.
```AgentChecker.waitUntilAgentReady(10, TimeUnit.SECONDS))```

## Debug
Various flags can be enabled to enable debugging

#### Solarwinds APM core logs
(WIP)
Various flags can be enabled to enable debugging. You can define the environment variable `SW_APM_DEBUG_LEVEL` to `debug` to print verbose logs. The OpenTelemetry core agent also provides an JVM argument `-Dotel.javaagent.debug=true` to enable detailed logs.

#### Muzzling
OT provides Muzzling which matches classes/fields/methods used by instrumentation vs the ones available on the running JVM. If there are any mismatch, the instrumentation will be silently disabled unless debugging flag such as below is provided in the JVM args:
Expand All @@ -94,10 +54,6 @@ Repackages the OT original agent with our custom compoenents (such as Sampler, T

This produces a new agent, that contains both the OT agent and our changes.

This approach is a middleground of the other 2 approaches:
1. AO as a pure OT extension, isolated from the OT agent - https://github.com/appoptics/appoptics-opentelemetry-java#opentelemetry-auto-agent
2. AO as a fork of OT java instrumentation agent - https://github.com/open-telemetry/opentelemetry-java-instrumentation

The advantage of this approach:
1. More control over the OT agent logic, for example we can have another layer of agent and modify the OT agent entry point by [changing the MANIFEST file](https://github.com/appoptics/opentelemetry-custom-distro/blob/master/agent/build.gradle#L48)
2. Since this is a separate repo from the OT java instrumentation, we have loose coupling here. Updates from OT java instrumentation and changes in this repo are less likely to have conflicts with eachother.
Expand All @@ -110,16 +66,7 @@ The disadvantage of this approach:
1. Whenever OT provides a newer version of agent, we will need to rebuild the agent on this repo too if we want the updates.

#### appoptics-opentelemetry-sdk
An SDK artifact that has the same interfaces as our existing appoptics SDK (https://librato.github.io/java-agent-sdk-javadoc/com/appoptics/api/ext/package-summary.html), under the hood, this translates SDK calls into OT operations (spans), therefore it can work with application that has existing manual OT SDK instrumentation.

In fact this artifact is currently not working on its own (missing dependency on `custom` and `core-bootstrap` in the built artifact). There is no strong use case for this artifact, the more useful case is addressed by `appoptics-opentelemetry-sdk-shaded` which is built on top of this artifact.

#### appoptics-opentelemetry-sdk-shaded
Repackage `appoptics-opentelemetry-sdk` to [shaded name space](https://github.com/appoptics/opentelemetry-custom-distro/blob/master/gradle/shadow.gradle), for example `io.opentelemetry.api` -> `io.opentelemetry.javaagent.shaded.io.opentelemetry.api` so SDK calls works with the shaded classes in OT auto agent.

This is expected to be used with the enhanced OT auto agent repackaged by our sub-project `agent` or the pure OT agent with our extension jar from sub-projec `sdk-extension`.

The use case would be someone using our existing AppOptics agent with SDK calls can migrate to use our enhanced OT agent with this `appoptics-opentelemetry-sdk-shaded` artifact. There should be no code change except changes to build process to reference this new artifact instead of the existing appoptics SDK
Currently it has only one API: the agent ready checker.

#### custom
Our main implementation for OT SPI - which scans implementation using Java service loader. We provide our "implementation" to various OT services such as Sampler, Tracer to enable various AO specific features - AO sampling, profiling, detailed trace reporting/export etc. This also contains various intiailization code and resource files such as default config, SSL cert for gRPC to collector etc.
Expand Down Expand Up @@ -148,21 +95,6 @@ so the muzzle plugin will inject our `Tracer` to the application classloader
A very good description of instrumentation with bytebuddy/OT can be found in [here](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/contributing/writing-instrumentation-module.md#advice-classes)


#### sdk-extensions
Repackages and builds the AO extension jar which runs with the original OT agent (vs the agent built from `agent` sub-project). Take note that since classes from this jar are appended via `-Dotel.javaagent.experimental.initializer.jar` which is loaded by application classloader (instead of OT agent classloader), no custom shadowing is performed (no putting in `inst` and renaming extension from `class` to `classdata`). Regular shadowing/shading is still applied (`io.opentelemetry.xyz` -> `io.opentelemetry.javaagent.shaded.io.opentelemetry.xyz`) as this is run with the OT original agent which some OT class references are still shaded. Take note that in order ot use this extension, it has to be augmented with `-Xbootclasspath` to append `sdk-extensions-bootstrap` jar to bootstrap class path (see [sdk-extensions-bootstrap](sdk-extensions-bootstrap) for details)

#### sdk-extensions-bootstrap
Repackages `core` and `metric` from joboe and append these classes to the bootstrap classpath. This is necessary as sdk-extension's instrumentation classes are injected into application classloader, these instrumentations reference our classes in joboe `core` and `metrics`. If the `core` classes are included in the sdk-extension via `-Dotel.javaagent.experimental.initializer.jar`, then they are unavailable to such classloader.

More details are documented in https://github.com/appoptics/appoptics-opentelemetry-java/pull/5 . Take note that since the instrumentations are now "muzzled", it no longer triggers the same exception as seen in the PR, but the instrumentation would still fail with similar classloading failure:
```
[otel.javaagent 2021-07-08 11:25:46:278 -0700] [http-nio-8080-exec-1] WARN muzzleMatcher - -- com.appoptics.opentelemetry.instrumentation.AoStatementTracer:22 Missing class com.tracelytics.util.BackTraceUtil
```

As discussed in https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/3350, one approach is to append the `core` and `metrics` classes to bootstrap classloader by using a separate jar, which is produced by this sub-project






Expand Down
3 changes: 2 additions & 1 deletion agent/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ dependencies {
customShadow project(path: ":custom", configuration: "shadow")
customShadow project(path: ":instrumentation", configuration: "shadow")
implementation project(path: ":core-bootstrap")
implementation project(path: ":appoptics-opentelemetry-sdk")
implementation "io.opentelemetry.javaagent:opentelemetry-javaagent:${versions.opentelemetryJavaagent}"
implementation "com.appoptics.agent.java:core:${versions.appopticsCore}"
implementation "com.appoptics.agent.java:metrics:${versions.appopticsMetrics}"
Expand All @@ -34,7 +35,7 @@ CopySpec isolateSpec() {
}

shadowJar {
classifier = null
archiveClassifier.set(null)
}

tasks {
Expand Down
50 changes: 0 additions & 50 deletions appoptics-opentelemetry-sdk-shaded/build.gradle

This file was deleted.

2 changes: 1 addition & 1 deletion appoptics-opentelemetry-sdk/build.gradle
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apply plugin: 'java'
apply plugin: "maven-publish"

group = "com.appoptics.agent.java"
group = "com.solarwinds.agent"

dependencies {
compileOnly("io.opentelemetry:opentelemetry-sdk:${versions.opentelemetry}")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
package com.appoptics.api.ext;

import com.appoptics.opentelemetry.extensions.initialize.Initializer;
import com.appoptics.opentelemetry.core.AgentState;
import io.opentelemetry.javaagent.bootstrap.AgentInitializer;

import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;
Expand All @@ -14,37 +13,29 @@
* @author pluk
*/
public class AgentChecker {
private static final Logger LOGGER = Logger.getLogger("appoptics-sdk");
private static final String APPOPTICS_SERVICE_KEY = "otel.appoptics.service.key";
private static final Logger LOGGER = Logger.getLogger("solarwinds-sdk");
private static boolean IS_EXTENSION_AVAILABLE = false;
private static final String SERVICE_KEY;

static {
String readServiceKey = null;
try {
Class.forName("com.appoptics.opentelemetry.extensions.initialize.Initializer");
Class.forName("com.appoptics.opentelemetry.core.AgentState");
setIsExtensionAvailable(true); //TODO version check?
readServiceKey = System.getProperty(APPOPTICS_SERVICE_KEY);
}
catch (ClassNotFoundException e) {
//perhaps running in OT agent environment, try agent classloader instead
try {
AgentInitializer.getAgentClassLoader().loadClass("com.appoptics.opentelemetry.extensions.initialize.Initializer");
AgentInitializer.getExtensionsClassLoader().loadClass("com.appoptics.opentelemetry.core.AgentState");
setIsExtensionAvailable(true); //TODO version check?
readServiceKey = System.getProperty(APPOPTICS_SERVICE_KEY);
}
catch (Throwable e2) {
LOGGER.log(Level.INFO, "AppOptics extensions not available");
LOGGER.log(Level.INFO, "Solarwinds APM extensions not available");
}
}
catch (NoClassDefFoundError e) {
/* This is not so expected as ClassLoader is supposed to throw ClassNotFoundException, but some loaders might throw NoClassDefFoundError instead */
LOGGER.log(Level.INFO, "AppOptics extensions not available");
LOGGER.log(Level.INFO, "Solarwinds APM extensions not available");
}
finally {
SERVICE_KEY = readServiceKey;
}

}


Expand All @@ -60,24 +51,8 @@ public class AgentChecker {
* @return whether the agent is ready
*/
public static boolean waitUntilAgentReady(long timeout, TimeUnit unit) {
if (isExtensionAvailable() && SERVICE_KEY != null) {
try {
//Future<?> future = Initializer.initialize(serviceKey);
final Future<?> future = Initializer.getStartupTasksFuture();
if (future != null) {
future.get(timeout, unit);
return true;
}
else {
LOGGER.warning("AppOptics can only be used with javaagent");
return false;
}

}
catch (Exception e) {
LOGGER.warning("Agent is still not ready after waiting for " + timeout + " " + unit);
return false;
}
if (isExtensionAvailable()) {
return AgentState.waitForReady(timeout, unit);
}
return false;
}
Expand Down

This file was deleted.

Loading

0 comments on commit 3b25110

Please sign in to comment.