Setting up PTP with multiple cameras

[canliu0414]

Describe what you want to implement and what the issue & the steps to reproduce it are:

Hi there,

I am trying to set up a system to synchronously record videos/frames from multiple cameras. My previous progress is in this post.

I currently hit two problems, which seems to be related.

1. I want to get the time stamps from each frame, which I am trying to achieve with the data chunk feature. However, the timestamp I get from there is tick number from camera power on, which is not the most informative. Ideally, it is in a format which I can interpret as relative to computer time (or to UTC time). It is currently not in UTC time because (in my understanding) I don't have a PTP grandmaster clock (GPS receiving equipment) in my system.

2. Currently, I think multiple cameras are being assigned as master, so synchronization is not guaranteed; surprisingly, sometimes synchronization still works even though there are multiple masters. I saw some instruction on setting up PTP daemon to organize the process, but only with Linux system (and also, the most recent post was not resolved yet). I am currently with a windows system, and I am a bit lost how to approach this. I also tried the teaming method to "combine" ethernet ports, but it didn't seem to work; or maybe I didn't do it correctly.

Any suggestions will be greatly apprecieated! Majority of my code is in the previous post too. Thank you!

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

OS: Microsoft Windows 11 Enterprise
System Type: x64-based PC
RAM: 128 GB

POE interface card: IOI 4-CH PCIE 2.5GIGABIT POE+ CARD
cables: CAT6 ethernet cable
cameras: a2A1920-51gmPRO, ac21920-51gcBAS

Runtime information:

python: 3.9.0 (default, Nov 15 2020, 08:30:55) [MSC v.1916 64 bit (AMD64)]
platform: win32/AMD64/10
pypylon: 3.0.1 / 7.4.0.38864

[canliu0414]

hi, a follow up on this:

PTP synchronization is working now by bridging the 4 ports (teaming them didn't work). I now have only one master and rest are slaves.

Regarding my timestamp question, how to get computer time/UTC time of frame acquisition (eg exposure timestamp) of each frame? Do I need to set up the ptp daemon on my computer to do that? and will doing that guarantee that the timestamps will be in UTC time, instead of tick numbers from camera power on?

Thank you!

[canliu0414]

Also, as I am trying to play with time stamp features, both GevTimestampControlReset and GevTimestampControlLatch command didn't work (Node not existing). What could be the reason?

[SMA2016a]

may be the node name has been renamed. Please check with pylonviewer first. the parameter search function of pylon viewer might very helpful for you.

[canliu0414]

an update on this: we decided to trigger frames by sending action commans, and take the computer time right before each action command sent, and recording this time as the "timestamp" of each frame. We compared this saved "timestamp" to time recorded from camera GPIO output time (linesource as exposure active). The difference between these two time are mostly below 20ms, which is goog enough for our purpose.

I think ideally, we find a way to sync our camera time to computer time, but we couldn't find any straightforward way. If there's any suggestions, please let me know.

[thiesmoeller]

System time to NIC time you have to use the tool phc2sys.

GevtimestampReset latch etc is not available in ptp mode.
It is of no need as the time of the camera is the time of the host

[canliu0414]

An update on this: so now, for our purpose of "knowing the timestamp of each frame", we are using action commands to trigger frames and record this action command time. However, I noticed that on documentation it says action command will execute on multiple cameras at "roughly the same time", so I guess the way to make them execute at exactly the same time will be through PTP so you make sure the action time means the same across all cameras.

Previously we had the problem that multiple cameras were master when trying to set up PTP; when we did the bridging of the ethernet port, this problem was solved that we have 1 master and rest are slaves. However, I just noticed that in my code I was not checking if each slave is "Locked" to master, and turns out their were not (PtpServoStatus is "unknown") so I guess PTP was still not synced sufficiently in the end. With this half failed PTP, if I point the two cameras to a milisecond clock, they still captured the same number, which seems to me they are still "synced" at least on ms level. I also tested with action command even without enabling PTP, and they were still "synced" on millisecond level.

so I think in terms of timestamp and ms level "sync", current set up seems to be ok for our purpose. But I am still curious what are the steps needed to really sync the multiple cameras with PTP. And please also let me know if you have any suggestions on our set up, thanks!

[PolymaTh-EE98]

If the cameras time is not the time from power on I believe it comes from PTP I’m not sure if anything else would update it.

[thiesmoeller]

The main problem with PTP on windows OS is, that there is no native support for a grand master clock with HW timestamps. Newer windows 10/11 versions have support for ptp slave clock.

The possible solutions are:

Bridging all Interfaces ( this will create a connected Ethernet between the interfaces and one of the cameras will be the Grand Master.

Connecting the cameras and the interfaces via an intermediate switch.

Licensing commercial PTP software from e.g. Meinberg.

Running ptp4l in WSL2 ( with proper configuration will allow to run the PC as grandmaster with synchronized clock over all Interfaces ) resulting clock accuracy will be lower due to the large amount of software layers involved.

[canliu0414]

Thanks for the suggestions. Because of limited time we haven't got chance to try these approaches. I have been trying the action command trigger + pylon viewer acquisition method. However, when I tested multiple cameras on our computers, it kept having errors and crash the recordings. The main two error messages include:

1. Type Time Source Message
   Error 2024-06-14 16:59:38.471 C02 (40482782) Image acquisition on "C02 (40482782)" failed! Error: "The buffer was incompletely grabbed. This can be caused by performance problems of the network hardware used, i.e. network adapter, switch, or ethernet cable. To fix this, try increasing the camera's Inter-Packet Delay in the Transport Layer category to reduce the required bandwidth, and adjust the camera's Packet Size setting to the highest supported frame size."

2. Cannot write video frame. The passed image is invalid. : InvalidArgumentException thrown (file 'videowriterimpl.cpp', line 407) : FatalMediaWriterException thrown (file 'VideoMediaWriterAdapter.cpp', line 97)

I am not sure what is the problem we are running into, maybe some network setting not optimized so our system is not giving the best performance? I have set jumbo frame to maximum, it seems to me inter-packet delay will decrease sampling rate so I didn't change that.

Could someone give some suggestions on where to look into for these two errors we are seeing? Thank you!

[SMA2016a]

the error message tells you what to do.

Error 2024-06-14 16:59:38.471 C02 (40482782) Image acquisition on "C02 (40482782)" failed! Error: "The buffer was incompletely grabbed. This can be caused by performance problems of the network hardware used, i.e. network adapter, switch, or ethernet cable. To fix this, try increasing the camera's Inter-Packet Delay in the Transport Layer category to reduce the required bandwidth, and adjust the camera's Packet Size setting to the highest supported frame size

please turn to your local support team to get support for system optimization. Nothing related to pypylon.