Null pointer exception when creating graphs

[stRodeo]

Hello,

I built the example using visual studio and everything compiles. However, when I launch the executable I have the following error
at the line 147 when graphs are created:

Exception thrown at 0x00007FFA5DCEA5B5 (libmp.dll) in TestMediaPipe.exe: 0xC0000005: Access violation reading location 0x0000000000000020.

There are no error from the console. It's just blank

I'm not really sure where to look and how to debug the following error. Help would be much appreciated!
Thanks a lot

[rajkundu]

Hi! Thank you for providing the console error text in your edit; that helped clue me in. I think the issue here is that you are literally passing the word "graph" instead of a MediaPipe/Protobuf graph configuration; what happens if you remove the quotes?

In this repo's example graph is a const char* defined as follows:

libmp-example/main.cpp

Lines 76 to 144 in d682b12

	const char* graph = R"(
	# MediaPipe graph that performs face mesh with TensorFlow Lite on CPU.
	# Input image. (ImageFrame)
	input_stream: "input_video"
	# Output image with rendered results. (ImageFrame)
	output_stream: "output_video"
	# Collection of detected/processed faces, each represented as a list of
	# landmarks. (std::vector<NormalizedLandmarkList>)
	output_stream: "multi_face_landmarks"
	# Throttles the images flowing downstream for flow control. It passes through
	# the very first incoming image unaltered, and waits for downstream nodes
	# (calculators and subgraphs) in the graph to finish their tasks before it
	# passes through another image. All images that come in while waiting are
	# dropped, limiting the number of in-flight images in most part of the graph to
	# 1. This prevents the downstream nodes from queuing up incoming images and data
	# excessively, which leads to increased latency and memory usage, unwanted in
	# real-time mobile applications. It also eliminates unnecessarily computation,
	# e.g., the output produced by a node may get dropped downstream if the
	# subsequent nodes are still busy processing previous inputs.
	node {
	calculator: "FlowLimiterCalculator"
	input_stream: "input_video"
	input_stream: "FINISHED:output_video"
	input_stream_info: {
	tag_index: "FINISHED"
	back_edge: true
	}
	output_stream: "throttled_input_video"
	}
	# Defines side packets for further use in the graph.
	node {
	calculator: "ConstantSidePacketCalculator"
	output_side_packet: "PACKET:0:num_faces"
	output_side_packet: "PACKET:1:use_prev_landmarks"
	output_side_packet: "PACKET:2:with_attention"
	node_options: {
	[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
	packet { int_value: 1 }
	packet { bool_value: true }
	packet { bool_value: true }
	}
	}
	}
	# Subgraph that detects faces and corresponding landmarks.
	node {
	calculator: "FaceLandmarkFrontCpu"
	input_stream: "IMAGE:throttled_input_video"
	input_side_packet: "NUM_FACES:num_faces"
	input_side_packet: "USE_PREV_LANDMARKS:use_prev_landmarks"
	input_side_packet: "WITH_ATTENTION:with_attention"
	output_stream: "LANDMARKS:multi_face_landmarks"
	output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
	output_stream: "DETECTIONS:face_detections"
	output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
	}
	# Subgraph that renders face-landmark annotation onto the input image.
	node {
	calculator: "FaceRendererCpu"
	input_stream: "IMAGE:throttled_input_video"
	input_stream: "LANDMARKS:multi_face_landmarks"
	input_stream: "NORM_RECTS:face_rects_from_landmarks"
	input_stream: "DETECTIONS:face_detections"
	output_stream: "IMAGE:output_video"
	}
	)";

[stRodeo]

Thanks a lot for the fast answer. Oops! Sorry I pasted the wrong error and corrected my comment as you answered it. Edited my original comment with proper error

[rajkundu]

No problem. I'm no expert, so I googled your access violation error. It seems that this occurs mostly when something which shouldn't be null is, in fact, null. This makes sense, as 0x0000000000000020 is very close to null and is almost certainly a bad address.

My guess is that mediapipe::LibMP::Create is failing and returning NULL, but this doesn't explicitly cause a crash because the example code in this repository doesn't do any error checking. So, I would start by teasing out the following line

libmp-example/main.cpp

Line 147 in d682b12

std::shared_ptr<mediapipe::LibMP> face_mesh(mediapipe::LibMP::Create(graph, "input_video"));

into separate parts:

mediapipe::LibMP* face_mesh = mediapipe::LibMP::Create(graph, "input_video"); // create LibMP object
std::cout << "LibMP object address: " << face_mesh << std::endl; // address SHOULD NOT be nullptr!

What do you see if you do this? What does your whole file look like, too? The more info and context you can provide, the better.

[stRodeo]

My cpp file essentially the same as main.cpp from the libmp example. I'm simply building using VS instead of cmake.

I replaced

libmp-example/main.cpp

Line 147 in d682b12

std::shared_ptr<mediapipe::LibMP> face_mesh(mediapipe::LibMP::Create(graph, "input_video"));

And now the same error is on this line that is creating the LibMP object

 mediapipe::LibMP* face_mesh = mediapipe::LibMP::Create(graph, "input_video"); // create LibMP object

I have copied the bazel-bin/mediapipe modules and models to the same folder as my executable as well as the openCV dll and downloaded the .tflites to my models folder. Could I be missing something regarding my folder structure?

Could I be using the wrong .tflites, the wrong version of mediapipe or the wrong version of protobuf?

[rajkundu]

I unfortunately haven't used anything other than CMake, so I'm not sure. Are you sure that your manual copies are identical to what CMake accomplishes in the following lines?

libmp-example/CMakeLists.txt

Lines 66 to 88 in d682b12

	### Post-build copies ###
	# LibMP - MediaPipe Shared Library
	add_custom_command(TARGET main POST_BUILD
	COMMAND ${CMAKE_COMMAND} -E copy "${LIBMP_SHARED_LIBRARY}" "$<TARGET_FILE_DIR:main>"
	COMMENT "Copying ${LIBMP_SHARED_LIBRARY}"
	)
	# Mediapipe Modules
	add_custom_command(TARGET main POST_BUILD
	COMMAND ${CMAKE_COMMAND} -E copy_directory "${MEDIAPIPE_DIR}/bazel-bin/mediapipe/modules" "$<TARGET_FILE_DIR:main>/mediapipe/modules"
	COMMENT "Copying mediapipe/modules"
	)
	# Mediapipe Models
	add_custom_command(TARGET main POST_BUILD
	COMMAND ${CMAKE_COMMAND} -E copy_directory "${MEDIAPIPE_DIR}/bazel-bin/mediapipe/models" "$<TARGET_FILE_DIR:main>/mediapipe/models"
	COMMENT "Copying mediapipe/models"
	)
	# OpenCV (Windows only)
	IF(WIN32)
	add_custom_command(TARGET main POST_BUILD
	COMMAND ${CMAKE_COMMAND} -E copy "${LIBMP_BIN_DIR}/opencv_world460.dll" "$<TARGET_FILE_DIR:main>"
	COMMENT "Copying OpenCV DLL (Release)"
	)
	ENDIF()

This is the folder structure of my build/Release folder after building with CMake:

• mediapipe/
  • models/
  • modules/
    • face_detection/
    • face_landmark/
    • hand_landmark/
    • holistic_landmark/
    • iris_landmark/
    • objectron/
    • palm_detection/
    • pose_detection/
    • pose_landmark/
    • selfie_segmentation/
• libmp.dll
• libprotobuf.lib
• main.exe
• MP_FORMATS.lib
• opencv_world460.dll

My mediapipe/models folder contains two .tflite files (hair_segmentation.tflite and ssdlite_object_detection.tflite), and each subfolder under mediapipe/modules contains many files. Is this consistent with your setup?

[donghyunismyname]

I had the same error message and I resolved it. In my case, the cause was that libmp.dll was complied the wrong way in the first place.

When complinging mediapipe using VS2022, the following error occurred

inference_calculator_options_registry.lo.lib(inference_calculator_options_lib.obj) : error LNK2019: unresolved external symbol

So I commented out some lines of the source code following the answer:
google-ai-edge/mediapipe#4060 (comment)

Then mediapipe complied successfully. However, as it turned out, the same runtime error as the original poster occurred.
I guess I commented out some crucial lines of mediapipe source code that prevented creating graph from mediapipe.

My solution was to complie libmp.dll with VS2019, not VS2022. For some reason that I cannot even imagine, mediapipe complies successfully with VS2019 but not with VS2022.

[donghyunismyname]

I'm not sure this is a proper place, but I want to express my gratitude to rajkundu, the author of this repository. Thank you so much. You are my savior. It would have been impossible to integrate MediaPipe to my project without help of this repository.

Just out of curiosity, how did you do that? Google does not provide any method to use MediaPipe as a C++ library. You can only use it with Bazel, which is extreamly inconvenient. A tutorial on MediaPipe I found on the internet says:

So the only way (at least for beginners) to use MP is to make your projects not only Bazel projects but parts of the MP project, located inside the mediapipe/ directory, just like MP examples. From our point of view, this is extremely ugly. And not using any IDE does not make coding in C++ any easier.

Read more at https://www.it-jim.com/blog/the-bizarre-google-world/

As I see it, it would take tremendus effort and expertise to build a C++ library out of MediaPipe by manually examining this monster, the orignal mediapipe project repository. The code structure is way too complex to comprehend, build script is involved with thousands of C++ sources. Google does not provide any appropriate tutorial. It is simply impossible for me. How did you do that?

[rajkundu]

@donghyunismyname That is very kind of you to say - thank you! However, pretty much all credit goes to @asprecic. He is the one who figured out how to compile MediaPipe into a shared library using Bazel and kindly shared libexample with us all (google-ai-edge/mediapipe#1162 (comment)). Everything I have done has been extremely simple in comparison - I just learned from, adapted, and expanded upon libexample to create LibMP.

I completely agree with you - MediaPipe is a monster, and I am a mere mortal like you. I honestly have no idea how @asprecic figured this all out, but I am just as thankful as you are that he did.

LibMP is far from perfect and quite hacky, but it filled enough holes for me to get my work done. I hope that it helps as many others as possible, and I do apologize (on behalf of Google, actually...) for the places that it falls short. In any case, thank you very, very much, @asprecic!! 😄

[stRodeo]

@rajkundu in order to narrow everything down, I rebuilt the libmp dll with the only difference being my python path in build_libmp_lib.bat, then I built the example the using cmake instead of VS. I also tried @donghyunismyname's suggestion to build libmp with VS 2019.

Everything builds like it did, and still get the same error/crash at the same line.

libmp-example/main.cpp

Line 147 in d682b12

std::shared_ptr<mediapipe::LibMP> face_mesh(mediapipe::LibMP::Create(graph, "input_video"));

Exception thrown at 0x00007FFA5DCEA5B5 (libmp.dll) in TestMediaPipe.exe: 0xC0000005: Access violation reading location 0x0000000000000020.

Is there any way I can enable more logs that can help me troubleshoot the error further?
Could it be possible to have access to an archive of your working project in order to find the differences with mine?
Thanks a lot!

[rajkundu]

@stRodeo I'm setting up a new machine from scratch for work, and I'm trying to build LibMP on it as well. I ran into google-ai-edge/mediapipe/issues/4060. Furthermore, I'm having the same crash as you when trying to create graphs. Perhaps this is related to that issue?

The main difference between machines is that I'm running the latest version of VS 2022 Build Tools on my new machine (14.5.0), whereas I had 14.3.5 on my old machine. I am not sure how to enable any extra logs, but I'll poke around and let you know what I find over the next few days.

[rajkundu]

@stRodeo This seems to be a VS 2022 17.4+ issue - see google-ai-edge/mediapipe#3851 and google-ai-edge/mediapipe#4060. Thank you very much, @donghyunismyname, for sharing google-ai-edge/mediapipe#4060 and helping us get to the bottom of this issue. As suggested, using older versions of VS work - even VS 2022 17.3.7 works, just not 17.4+. The current best workaround, in my opinion, is to use an older version of VS. If I were more skilled at C++, perhaps I could figure out how to work around MediaPipe's use of static data member templates, but I unfortunately am not.

@donghyunismyname @stRodeo I have filed an issue with Microsoft here. If you can, please consider upvoting the issue on Microsoft's forum to ensure that we can build MediaPipe with future versions of VS!

[rajkundu]

I can confirm that the underlying bug in VS 2022 and has been fixed in version 17.6 Preview 1. So, the permanent solution is to avoid using VS 2022 17.4.x & 17.5.x and either use VS 2022 <=17.3 or >=17.6 upon release.