Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(analytical): Fix problem of running Giraph apps with GraphScope Deployed on k8s cluster #3900

Draft
wants to merge 55 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
68ebe69
todo: test user-writable
zhanglei1949 May 29, 2024
355cecd
testing
zhanglei1949 Jun 3, 2024
e344419
todo: resolve edata
zhanglei1949 Jun 3, 2024
542edd3
use baseTypedArray
zhanglei1949 Jun 3, 2024
86bccd2
fix
zhanglei1949 Jun 3, 2024
0be8774
Todo: solve python entrance problem
zhanglei1949 Jun 3, 2024
4e48e22
code minor refine
zhanglei1949 Jun 4, 2024
bb02045
minor
zhanglei1949 Jun 4, 2024
6d30765
reverts changes on CMakeLists
zhanglei1949 Jun 5, 2024
f457e3d
fix tests
zhanglei1949 Jun 5, 2024
69ecac9
format
zhanglei1949 Jun 5, 2024
ec92cf3
remove test script
zhanglei1949 Jun 5, 2024
44d57c7
format
zhanglei1949 Jun 5, 2024
418dd71
Merge branch 'main' into giraph-user-writable
zhanglei1949 Jun 5, 2024
2bbcb2d
cpp format
zhanglei1949 Jun 5, 2024
a76c3b6
Merge branch 'main' into giraph-user-writable
zhanglei1949 Jun 5, 2024
8cc6860
add python test
zhanglei1949 Jun 7, 2024
99bb1aa
fix dockerfile and precompile.py
zhanglei1949 Jun 5, 2024
8ab68c8
able to run
zhanglei1949 Jun 7, 2024
0b81735
fix
zhanglei1949 Jun 11, 2024
1ee9c3b
we can't assume edata/vdata must be string
zhanglei1949 Jun 11, 2024
eb0454c
fix edge input format
zhanglei1949 Jun 11, 2024
4566508
minor fix
zhanglei1949 Jun 11, 2024
9e2614d
fixed
zhanglei1949 Jun 11, 2024
3c8a93e
add template specification
zhanglei1949 Jun 11, 2024
4e430df
add more template specialization
zhanglei1949 Jun 11, 2024
5de1531
for user writable, output with toString method
zhanglei1949 Jun 17, 2024
8d51d1e
todo: impl hdfs reader
zhanglei1949 Jun 21, 2024
4ed32fe
impl the file loader
zhanglei1949 Jun 24, 2024
aeb2da9
move to giraph sdk
zhanglei1949 Jun 24, 2024
bb579ba
refactor loader
zhanglei1949 Jun 24, 2024
7296fd4
update to 2.10.2
zhanglei1949 Jun 25, 2024
c5bf0d4
add dummy impl
zhanglei1949 Jun 25, 2024
e122746
try fix waiting
zhanglei1949 Jun 25, 2024
73de18f
try to catch error
zhanglei1949 Jun 25, 2024
11fcd4b
support reading from hdfs
zhanglei1949 Jun 26, 2024
871a912
fix reading from hdfs and build image
zhanglei1949 Jul 3, 2024
9630ab0
impl circel
zhanglei1949 Jul 12, 2024
87453fe
stash
zhanglei1949 Jul 12, 2024
a4edc7f
fixing circlePIE
zhanglei1949 Jul 13, 2024
6727e98
add annotation invoker
zhanglei1949 Jul 13, 2024
0c0ce2c
fix new example
zhanglei1949 Jul 14, 2024
0e1e983
run on k8s
zhanglei1949 Jul 15, 2024
0e20b14
use availabel memory
zhanglei1949 Jul 23, 2024
5f36aa5
parallel app base
zhanglei1949 Jul 23, 2024
dda29c4
merge main
zhanglei1949 Jul 29, 2024
dcae892
fix compilation
zhanglei1949 Aug 1, 2024
5a7cbe4
add method assign
zhanglei1949 Aug 26, 2024
64a32e4
fix memory leak
zhanglei1949 Aug 26, 2024
70d37b5
add circle parallel and try to debug memory increasing
zhanglei1949 Aug 29, 2024
249cb48
impl message app
zhanglei1949 Aug 29, 2024
3b52a4a
reproceduce the memory leak
zhanglei1949 Sep 2, 2024
87a50e9
don't try to interpret the received message
zhanglei1949 Sep 2, 2024
db3162f
todo: impl project adjlist
zhanglei1949 Sep 2, 2024
80bbd18
todo: test new ProjectedAdjList
zhanglei1949 Sep 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 14 additions & 23 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,20 +1,18 @@

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/javascript-node
{
"name": "GraphScope",
"name": "GraphScope-Giraph",
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-dev:latest",

// Features to add to the dev container. More info: https://containers.dev/features.
"features": {
"ghcr.io/devcontainers/features/common-utils:2":{
"installZsh": "true",
"features": {
"ghcr.io/devcontainers/features/common-utils:2": {
"installZsh": "true",
"configureZshAsDefaultShell": "true",
"installOhMyZsh": true,
"upgradePackages": "false"
}
},
"upgradePackages": "false"
}
},
// Configure tool-specific properties.
"customizations": {
// Configure properties specific to VS Code.
Expand All @@ -28,20 +26,15 @@
]
}
},

// Set `remoteUser` to `root` to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "graphscope",

// Use 'postCreateCommand' to run commands before the container is created.
"initializeCommand": "sudo docker pull registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-dev:latest",

// Uncomment this to enable C++ and Rust debugging in containers
// "capAdd": ["SYS_PTRACE"],
// "securityOpt": ["seccomp=unconfined"],

// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [3000],

// Use 'portsAttributes' to set default properties for specific forwarded ports.
// More info: https://containers.dev/implementors/json_reference/#port-attributes
// "portsAttributes": {
Expand All @@ -50,23 +43,21 @@
// "onAutoForward": "notify"
// }
// },

// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "yarn install"

// Improve performance

// Uncomment these to mount a folder to a volume
// https://code.visualstudio.com/remote/advancedcontainers/improve-performance#_use-a-targeted-named-volume
// "mounts": [
// "source=${localWorkspaceFolderBasename}-node_modules,target=${containerWorkspaceFolder}/node_modules,type=volume"
// ],
"mounts": [
"source=/mnt/zhanglei/dev_mounts,target=/home/graphscope/dev_mounts,type=bind,consistency=cached"
],
"runArgs": [
"--network=host",
],
// "postCreateCommand": "sudo chown graphscope node_modules"


// Uncomment these to use a named volume for your entire source tree
// https://code.visualstudio.com/remote/advancedcontainers/improve-performance#_use-a-named-volume-for-your-entire-source-tree
// "workspaceMount": "source=gs,target=/workspaces,type=volume",
// "workspaceFolder": "/workspaces"
"postCreateCommand": "sudo chown -R graphscope /workspaces && bash pre-commit/install-hook.sh && bash pre-commit/prepare-commit-msg"
}
}
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.28.0
0.27.0
136 changes: 68 additions & 68 deletions analytical_engine/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -354,63 +354,63 @@ endif ()

# An executable workaround for graphx pregel.
if (ENABLE_JAVA_SDK)
add_executable(graphx_runner core/java/graphx_runner.cc core/java/javasdk.cc)
target_include_directories(graphx_runner PRIVATE core utils apps)
target_compile_definitions(graphx_runner PUBLIC ENABLE_JAVA_SDK)
target_link_libraries(graphx_runner PRIVATE ${CMAKE_DL_LIBS} gs_proto ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GFLAGS_LIBRARIES} ${JNI_LIBRARIES})

if (${LIBUNWIND_FOUND})
target_link_libraries(graphx_runner PRIVATE ${LIBUNWIND_LIBRARIES})
endif()
# add_executable(graphx_runner core/java/graphx_runner.cc core/java/javasdk.cc)
# target_include_directories(graphx_runner PRIVATE core utils apps)
# target_compile_definitions(graphx_runner PUBLIC ENABLE_JAVA_SDK)
# target_link_libraries(graphx_runner PRIVATE ${CMAKE_DL_LIBS} gs_proto ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GFLAGS_LIBRARIES} ${JNI_LIBRARIES})

# if (${LIBUNWIND_FOUND})
# target_link_libraries(graphx_runner PRIVATE ${LIBUNWIND_LIBRARIES})
# endif()
endif()


# Test targets
if (BUILD_TESTS)
add_executable(run_app test/run_app.cc core/object/dynamic.cc)
target_include_directories(run_app PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps utils apps)
target_link_libraries(run_app ${LIBGRAPELITE_LIBRARIES} ${GFLAGS_LIBRARIES} ${CMAKE_DL_LIBS} ${Boost_LIBRARIES} ${VINEYARD_LIBRARIES})
target_link_libraries(run_app OpenMP::OpenMP_CXX)
# add_executable(run_app test/run_app.cc core/object/dynamic.cc)
# target_include_directories(run_app PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps utils apps)
# target_link_libraries(run_app ${LIBGRAPELITE_LIBRARIES} ${GFLAGS_LIBRARIES} ${CMAKE_DL_LIBS} ${Boost_LIBRARIES} ${VINEYARD_LIBRARIES})
# target_link_libraries(run_app OpenMP::OpenMP_CXX)

if (ENABLE_JAVA_SDK)
add_executable(run_java_app test/run_java_app.cc core/java/javasdk.cc core/object/dynamic.cc)
target_include_directories(run_java_app PRIVATE core utils apps)
target_compile_definitions(run_java_app PUBLIC ENABLE_JAVA_SDK)
target_link_libraries(run_java_app gs_proto ${GFLAGS_LIBRARIES} ${MPI_CXX_LIBRARIES} ${JNI_LIBRARIES} ${Boost_LIBRARIES}
${CMAKE_DL_LIBS} ${GLOG_LIBRARIES} ${VINEYARD_LIBRARIES})
${CMAKE_DL_LIBS} ${GLOG_LIBRARIES} ${VINEYARD_LIBRARIES} ${JAVA_JVM_LIBRARY})

# java app benchmark
add_executable(property_graph_java_app_benchmarks benchmarks/property_graph_java_app_benchmarks.cc core/java/javasdk.cc core/object/dynamic.cc)
target_include_directories(property_graph_java_app_benchmarks PRIVATE core utils apps)
target_compile_definitions(property_graph_java_app_benchmarks PUBLIC ENABLE_JAVA_SDK)
target_link_libraries(property_graph_java_app_benchmarks gs_proto ${JNI_LIBRARIES} ${Boost_LIBRARIES} ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES})
# # java app benchmark
# add_executable(property_graph_java_app_benchmarks benchmarks/property_graph_java_app_benchmarks.cc core/java/javasdk.cc core/object/dynamic.cc)
# target_include_directories(property_graph_java_app_benchmarks PRIVATE core utils apps)
# target_compile_definitions(property_graph_java_app_benchmarks PUBLIC ENABLE_JAVA_SDK)
# target_link_libraries(property_graph_java_app_benchmarks gs_proto ${JNI_LIBRARIES} ${Boost_LIBRARIES} ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES})

# giraph runner
# # giraph runner
add_executable(giraph_runner test/giraph_runner.cc core/java/javasdk.cc)
target_include_directories(giraph_runner PRIVATE core utils apps)
target_compile_definitions(giraph_runner PUBLIC ENABLE_JAVA_SDK)
target_link_libraries(giraph_runner ${CMAKE_DL_LIBS} gs_proto ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GFLAGS_LIBRARIES} ${JNI_LIBRARIES})

# graphx related test
add_executable(projected_fragment_mapper_test test/projected_fragment_mapper_test.cc)
target_include_directories(projected_fragment_mapper_test PRIVATE core utils apps)
target_link_libraries(projected_fragment_mapper_test ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GLOG_LIBRARIES} ${GFLAGS_LIBRARIES})
# # graphx related test
# add_executable(projected_fragment_mapper_test test/projected_fragment_mapper_test.cc)
# target_include_directories(projected_fragment_mapper_test PRIVATE core utils apps)
# target_link_libraries(projected_fragment_mapper_test ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GLOG_LIBRARIES} ${GFLAGS_LIBRARIES})

add_executable(graphx_loader_test test/graphx_loader_test.cc)
target_include_directories(graphx_loader_test PRIVATE core utils apps)
target_compile_definitions(graphx_loader_test PUBLIC ENABLE_JAVA_SDK)
target_link_libraries(graphx_loader_test ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GLOG_LIBRARIES} ${GFLAGS_LIBRARIES})
# add_executable(graphx_loader_test test/graphx_loader_test.cc)
# target_include_directories(graphx_loader_test PRIVATE core utils apps)
# target_compile_definitions(graphx_loader_test PUBLIC ENABLE_JAVA_SDK)
# target_link_libraries(graphx_loader_test ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GLOG_LIBRARIES} ${GFLAGS_LIBRARIES})

add_executable(run_java_string_app test/run_java_string_app.cc core/java/javasdk.cc)
target_include_directories(run_java_string_app PRIVATE core utils apps)
target_compile_definitions(run_java_string_app PUBLIC ENABLE_JAVA_SDK)
target_link_libraries(run_java_string_app ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GLOG_LIBRARIES} ${GFLAGS_LIBRARIES} ${JNI_LIBRARIES})
# add_executable(run_java_string_app test/run_java_string_app.cc core/java/javasdk.cc)
# target_include_directories(run_java_string_app PRIVATE core utils apps)
# target_compile_definitions(run_java_string_app PUBLIC ENABLE_JAVA_SDK)
# target_link_libraries(run_java_string_app ${CMAKE_DL_LIBS} ${VINEYARD_LIBRARIES} ${Boost_LIBRARIES} ${GLOG_LIBRARIES} ${GFLAGS_LIBRARIES} ${JNI_LIBRARIES})

if (${LIBUNWIND_FOUND})
target_link_libraries(run_java_app ${LIBUNWIND_LIBRARIES})
target_link_libraries(property_graph_java_app_benchmarks ${LIBUNWIND_LIBRARIES})
# target_link_libraries(property_graph_java_app_benchmarks ${LIBUNWIND_LIBRARIES})
target_link_libraries(giraph_runner ${LIBUNWIND_LIBRARIES})
target_link_libraries(projected_fragment_mapper_test ${LIBUNWIND_LIBRARIES})
# target_link_libraries(projected_fragment_mapper_test ${LIBUNWIND_LIBRARIES})
endif ()
endif()

Expand All @@ -428,43 +428,43 @@ if (BUILD_TESTS)
target_link_libraries(${target} OpenMP::OpenMP_CXX)
endmacro()

add_vineyard_app(run_vy_app SRCS test/run_vy_app.cc)
add_vineyard_app(run_vy_app_compact SRCS test/run_vy_app_compact.cc)
add_vineyard_app(run_vy_app_local_vm SRCS test/run_vy_app_local_vm.cc)
# add_vineyard_app(run_vy_app SRCS test/run_vy_app.cc)
# add_vineyard_app(run_vy_app_compact SRCS test/run_vy_app_compact.cc)
# add_vineyard_app(run_vy_app_local_vm SRCS test/run_vy_app_local_vm.cc)

add_vineyard_app(run_load_from_stream SRCS test/run_load_from_stream.cc)
# add_vineyard_app(run_load_from_stream SRCS test/run_load_from_stream.cc)

add_vineyard_app(run_vy_ldbc SRCS test/run_vy_ldbc.cc)
target_include_directories(run_vy_ldbc PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
# add_vineyard_app(run_vy_ldbc SRCS test/run_vy_ldbc.cc)
# target_include_directories(run_vy_ldbc PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)

add_vineyard_app(run_ctx SRCS test/run_ctx.cc)
target_include_directories(run_ctx PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
target_link_libraries(run_ctx gs_proto)
# add_vineyard_app(run_ctx SRCS test/run_ctx.cc)
# target_include_directories(run_ctx PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
# target_link_libraries(run_ctx gs_proto)

add_vineyard_app(run_property_ctx SRCS test/run_property_ctx.cc)
target_include_directories(run_property_ctx PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
target_link_libraries(run_property_ctx gs_proto)
# add_vineyard_app(run_property_ctx SRCS test/run_property_ctx.cc)
# target_include_directories(run_property_ctx PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
# target_link_libraries(run_property_ctx gs_proto)

add_vineyard_app(run_pregel_app SRCS test/run_pregel_app.cc)
# add_vineyard_app(run_pregel_app SRCS test/run_pregel_app.cc)

add_vineyard_app(run_string_oid SRCS test/run_string_oid.cc)
target_include_directories(run_string_oid PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
# add_vineyard_app(run_string_oid SRCS test/run_string_oid.cc)
# target_include_directories(run_string_oid PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)

add_vineyard_app(run_empty_property SRCS test/run_empty_property.cc)
target_include_directories(run_empty_property PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)
# add_vineyard_app(run_empty_property SRCS test/run_empty_property.cc)
# target_include_directories(run_empty_property PRIVATE ${LIBGRAPELITE_INCLUDE_DIRS}/grape/analytical_apps)

add_vineyard_app(test_project_string SRCS test/test_project_string.cc)
# add_vineyard_app(test_project_string SRCS test/test_project_string.cc)

add_vineyard_app(basic_graph_benchmarks SRCS benchmarks/basic_graph_benchmarks.cc)
# add_vineyard_app(basic_graph_benchmarks SRCS benchmarks/basic_graph_benchmarks.cc)

add_vineyard_app(property_graph_loader SRCS benchmarks/property_graph_loader.cc)
# add_vineyard_app(property_graph_loader SRCS benchmarks/property_graph_loader.cc)

add_vineyard_app(property_graph_benchmarks SRCS benchmarks/property_graph_benchmarks.cc)
# add_vineyard_app(property_graph_benchmarks SRCS benchmarks/property_graph_benchmarks.cc)

add_vineyard_app(projected_graph_benchmarks SRCS benchmarks/projected_graph_benchmarks.cc)
# add_vineyard_app(projected_graph_benchmarks SRCS benchmarks/projected_graph_benchmarks.cc)

if (NETWORKX)
add_vineyard_app(test_convert SRCS test/test_convert.cc)
# add_vineyard_app(test_convert SRCS test/test_convert.cc)
endif ()
endif ()

Expand Down Expand Up @@ -509,17 +509,17 @@ if(ENABLE_JAVA_SDK)
set(GAE_JAVA_JNI_LIB "${GAE_JAVA_DIR}/grape-runtime/target/native/libgrape-jni.so")
endif()

add_custom_command(
OUTPUT "${GAE_JAVA_RUNTIME_JAR}"
COMMAND mvn -Drevision=${GRAPHSCOPE_ANALYTICAL_JAR_VERSION} clean install -DskipTests --quiet
DEPENDS gs_proto
WORKING_DIRECTORY ${GAE_JAVA_DIR}
COMMENT "Building GAE-java..."
VERBATIM
)
add_custom_target(grape_jni ALL
DEPENDS "${GAE_JAVA_RUNTIME_JAR}"
)
# add_custom_command(
# OUTPUT "${GAE_JAVA_RUNTIME_JAR}"
# COMMAND mvn -Drevision=${GRAPHSCOPE_ANALYTICAL_JAR_VERSION} clean install -DskipTests --quiet
# DEPENDS gs_proto
# WORKING_DIRECTORY ${GAE_JAVA_DIR}
# COMMENT "Building GAE-java..."
# VERBATIM
# )
# add_custom_target(grape_jni ALL
# DEPENDS "${GAE_JAVA_RUNTIME_JAR}"
# )
install(FILES DESTINATION lib)
install(FILES "${GAE_JAVA_RUNTIME_JAR}" DESTINATION lib)
install(FILES "${GAE_JAVA_GRAPHX_JAR}" DESTINATION lib)
Expand Down Expand Up @@ -578,7 +578,7 @@ install_gsa_binary(gs_proto)
install_gsa_binary(gs_util)

if (ENABLE_JAVA_SDK)
install_gsa_binary(graphx_runner)
# install_gsa_binary(graphx_runner)
endif()

install_gsa_headers("${PROJECT_SOURCE_DIR}/apps")
Expand Down
25 changes: 23 additions & 2 deletions analytical_engine/core/context/java_context_base.h
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,29 @@ class JavaContextBase : public grape::ContextBase {
}
JNIEnvMark m;
if (m.env()) {
m.env()->DeleteGlobalRef(url_class_loader_object_);
VLOG(1) << "Delete URL class loader";
// Delete the java objects
if (app_object_) {
m.env()->DeleteGlobalRef(app_object_);
VLOG(1) << "Delete app object";
}
if (context_object_) {
m.env()->DeleteGlobalRef(context_object_);
VLOG(1) << "Delete context object";
}
if (fragment_object_) {
m.env()->DeleteGlobalRef(fragment_object_);
VLOG(1) << "Delete fragment object";
}
if (mm_object_) {
m.env()->DeleteGlobalRef(mm_object_);
VLOG(1) << "Delete message manager object";
}
if (url_class_loader_object_) {
m.env()->DeleteGlobalRef(url_class_loader_object_);
VLOG(1) << "Delete url class loader object";
}
// Invoke a gc, since we have created a lot of objects.
InvokeGC(m.env());
} else {
LOG(ERROR) << "JNI env not available.";
}
Expand Down
38 changes: 21 additions & 17 deletions analytical_engine/core/java/java_loader_invoker.h
Original file line number Diff line number Diff line change
Expand Up @@ -137,14 +137,15 @@ boost::leaf::result<void> BuildArray(
return {};
}

static constexpr const char* JAVA_LOADER_CLASS =
"com/alibaba/graphscope/loader/impl/FileLoader";
static constexpr const char* JAVA_LOADER_CREATE_METHOD = "create";
static constexpr const char* JAVA_LOADER_FACTORY_CLASS =
"com/alibaba/graphscope/loader/LoaderFactory";
static constexpr const char* JAVA_BASE_LOADER_CLASS =
"com/alibaba/graphscope/loader/LoaderBase";
static constexpr const char* JAVA_LOADER_CREATE_METHOD = "createLoader";
static constexpr const char* JAVA_LOADER_CREATE_SIG =
"(Ljava/net/URLClassLoader;)Lcom/alibaba/graphscope/loader/impl/"
"FileLoader;";
static constexpr const char* JAVA_LOADER_LOAD_VE_METHOD =
"loadVerticesAndEdges";
"(ILjava/net/"
"URLClassLoader;)Lcom/alibaba/graphscope/loader/LoaderBase;";
static constexpr const char* JAVA_LOADER_LOAD_VE_METHOD = "loadVertices";
static constexpr const char* JAVA_LOADER_LOAD_VE_SIG =
"(Ljava/lang/String;Ljava/lang/String;)I";
static constexpr const char* JAVA_LOADER_LOAD_E_METHOD = "loadEdges";
Expand Down Expand Up @@ -389,16 +390,19 @@ class JavaLoaderInvoker {
gs::JNIEnvMark m;
if (m.env()) {
JNIEnv* env = m.env();
jclass loader_class =
LoadClassWithClassLoader(env, gs_class_loader_obj, JAVA_LOADER_CLASS);
CHECK_NOTNULL(loader_class);
jclass factory_class = LoadClassWithClassLoader(
env, gs_class_loader_obj, JAVA_LOADER_FACTORY_CLASS);
jclass base_loader_class = LoadClassWithClassLoader(
env, gs_class_loader_obj, JAVA_BASE_LOADER_CLASS);
CHECK_NOTNULL(factory_class);
CHECK_NOTNULL(base_loader_class);
// construct java loader obj.
jmethodID create_method = env->GetStaticMethodID(
loader_class, JAVA_LOADER_CREATE_METHOD, JAVA_LOADER_CREATE_SIG);
factory_class, JAVA_LOADER_CREATE_METHOD, JAVA_LOADER_CREATE_SIG);
CHECK(create_method);

java_loader_obj = env->NewGlobalRef(env->CallStaticObjectMethod(
loader_class, create_method, gs_class_loader_obj));
factory_class, create_method, worker_id_, gs_class_loader_obj));

if (env->ExceptionCheck()) {
env->ExceptionDescribe();
Expand All @@ -409,7 +413,7 @@ class JavaLoaderInvoker {
CHECK(java_loader_obj);

jmethodID loader_method = env->GetMethodID(
loader_class, JAVA_LOADER_INIT_METHOD, JAVA_LOADER_INIT_SIG);
base_loader_class, JAVA_LOADER_INIT_METHOD, JAVA_LOADER_INIT_SIG);
CHECK_NOTNULL(loader_method);

env->CallVoidMethod(java_loader_obj, loader_method, worker_id_,
Expand Down Expand Up @@ -524,8 +528,8 @@ class JavaLoaderInvoker {
gs::JNIEnvMark m;
if (m.env()) {
JNIEnv* env = m.env();
jclass loader_class =
LoadClassWithClassLoader(env, gs_class_loader_obj, JAVA_LOADER_CLASS);
jclass loader_class = LoadClassWithClassLoader(env, gs_class_loader_obj,
JAVA_BASE_LOADER_CLASS);
CHECK_NOTNULL(loader_class);

jmethodID loader_method = env->GetMethodID(
Expand Down Expand Up @@ -560,8 +564,8 @@ class JavaLoaderInvoker {
gs::JNIEnvMark m;
if (m.env()) {
JNIEnv* env = m.env();
jclass loader_class =
LoadClassWithClassLoader(env, gs_class_loader_obj, JAVA_LOADER_CLASS);
jclass loader_class = LoadClassWithClassLoader(env, gs_class_loader_obj,
JAVA_BASE_LOADER_CLASS);
CHECK_NOTNULL(loader_class);

jmethodID loader_method = env->GetMethodID(
Expand Down
Loading
Loading