Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High memory usage #34

Closed
zenas1210 opened this issue Jan 11, 2025 · 8 comments · Fixed by #35
Closed

High memory usage #34

zenas1210 opened this issue Jan 11, 2025 · 8 comments · Fixed by #35

Comments

@zenas1210
Copy link

Q A
Version 0.17.0

Support Question

Hi,
I've encountered some memory usage issues when using this with phpro/soap-client v4. I'm running out of memory when decoding certain responses, which aren't really big and php-soap/ext-soap-engine with phpro/soap-client v3.4 has no problem with them. Since the SOAP webservice I'm integrating with is private I can't share an end to end example of my problem. Is there any automated test or maybe just a code snippet which would enable me to demonstrate the issue by supplying the WSDL and a somewhat fake response with confidential data removed from it?
@veewee
Copy link
Member

veewee commented Jan 11, 2025 •
edited
Loading

We'll at least need a memory-problematic payload and wsdl to reproduce and start improving upon. The data inside it may be fake.

Is this happening during encoding, decoding or both? This might require 2 payloads then: a php objects representation for encoding and an XML payload for decoding.

If that's not possible, you could also do the profiling of the code yourself to see if there are abvious resources eating up memory. We're open for PRs.

@zenas1210
Copy link
Author

This happens only during decoding.
From first glance ObjectEncoder::iso seems to be called many times, probably too many, for properties which are not part of the response being decoded.
image

WSDL: https://fpy.noc.vmc.navblue.cloud/raidoapi/nocsoapapi.asmx?wsdl
Function: GetFlights
Response with some scalar values changed, 4GB of RAM couldn't handle it:

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <soap:Body>
      <GetFlightsResponse xmlns="http://raido.aviolinx.com/api/">
         <GetFlightsResult>
            <Errors />
            <Warnings />
            <StatusCode>200</StatusCode>
            <Result>
               <Flight>
                  <UniqueId>1</UniqueId>
                  <AirlineCode>AA</AirlineCode>
                  <ICAOAirlineCode>AAA</ICAOAirlineCode>
                  <FlightNumber>1</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T14:45:00</STD>
                  <STA>2025-01-11T19:40:00</STA>
                  <AircraftRegistration>LYAAA</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>60</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>0</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>3270</ScheduleGCDKM>
                  <ScheduleGCDNM>1766</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>2</UniqueId>
                  <AirlineCode>BB</AirlineCode>
                  <ICAOAirlineCode>BBB</ICAOAirlineCode>
                  <FlightNumber>2</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:00:00</STD>
                  <STA>2025-01-11T19:10:00</STA>
                  <AircraftRegistration>LYBBB</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>60</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>0</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>2644</ScheduleGCDKM>
                  <ScheduleGCDNM>1428</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>3</UniqueId>
                  <AirlineCode>CC</AirlineCode>
                  <ICAOAirlineCode>CCC</ICAOAirlineCode>
                  <FlightNumber>3</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:20:00</STD>
                  <STA>2025-01-11T22:00:00</STA>
                  <AircraftRegistration>TFPLA</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Canceled</Status>
                  <Type>A</Type>
                  <Canceled>true</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <IROPSParentId>0</IROPSParentId>
                  <ASMReasons>EQUI</ASMReasons>
                  <Complement>321LR</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4495</ScheduleGCDKM>
                  <ScheduleGCDNM>2427</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>4</UniqueId>
                  <AirlineCode>DD</AirlineCode>
                  <ICAOAirlineCode>DDD</ICAOAirlineCode>
                  <FlightNumber>4</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:20:00</STD>
                  <STA>2025-01-11T22:00:00</STA>
                  <AircraftRegistration>LYDDD</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4436</ScheduleGCDKM>
                  <ScheduleGCDNM>2395</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>5</UniqueId>
                  <AirlineCode>EE</AirlineCode>
                  <ICAOAirlineCode>EEE</ICAOAirlineCode>
                  <FlightNumber>5</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:50:00</STD>
                  <STA>2025-01-11T20:25:00</STA>
                  <AircraftRegistration>LYEEE</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>A</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>60</ArrivalLocalTimeDiff>
                  <IROPSParentId>0</IROPSParentId>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>3208</ScheduleGCDKM>
                  <ScheduleGCDNM>1732</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>6</UniqueId>
                  <AirlineCode>FF</AirlineCode>
                  <ICAOAirlineCode>FFF</ICAOAirlineCode>
                  <FlightNumber>6</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:50:00</STD>
                  <STA>2025-01-11T21:35:00</STA>
                  <AircraftRegistration>LYFFF</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>0</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4020</ScheduleGCDKM>
                  <ScheduleGCDNM>2170</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>7</UniqueId>
                  <AirlineCode>GG</AirlineCode>
                  <ICAOAirlineCode>GGG</ICAOAirlineCode>
                  <FlightNumber>7</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T15:50:00</STD>
                  <STA>2025-01-11T22:10:00</STA>
                  <AircraftRegistration />
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Canceled</Status>
                  <Type>A</Type>
                  <Canceled>true</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons>COMM</ASMReasons>
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4230</ScheduleGCDKM>
                  <ScheduleGCDNM>2284</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>8</UniqueId>
                  <AirlineCode>HH</AirlineCode>
                  <ICAOAirlineCode>HHH</ICAOAirlineCode>
                  <FlightNumber>8</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T16:00:00</STD>
                  <STA>2025-01-11T22:00:00</STA>
                  <AircraftRegistration>LYHHH</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>S</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>3874</ScheduleGCDKM>
                  <ScheduleGCDNM>2092</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
               <Flight>
                  <UniqueId>9</UniqueId>
                  <AirlineCode>II</AirlineCode>
                  <ICAOAirlineCode>III</ICAOAirlineCode>
                  <FlightNumber>9</FlightNumber>
                  <Suffix />
                  <DepartureAirportCode>VNO</DepartureAirportCode>
                  <ArrivalAirportCode>AMS</ArrivalAirportCode>
                  <DiversionAirportCode />
                  <FlightDate>2025-01-11T00:00:00</FlightDate>
                  <STD>2025-01-11T16:00:00</STD>
                  <STA>2025-01-11T22:05:00</STA>
                  <AircraftRegistration>LYIII</AircraftRegistration>
                  <ServiceTypeCode>J</ServiceTypeCode>
                  <OnwardAirlineCode />
                  <OnwardFlightNumber>0</OnwardFlightNumber>
                  <OnwardSuffix />
                  <Status>Airborne</Status>
                  <Type>A</Type>
                  <Canceled>false</Canceled>
                  <DepartureLocalTimeDiff>0</DepartureLocalTimeDiff>
                  <ArrivalLocalTimeDiff>-300</ArrivalLocalTimeDiff>
                  <ASMReasons />
                  <Complement>321S</Complement>
                  <CO2>0</CO2>
                  <ScheduleGCDKM>4100</ScheduleGCDKM>
                  <ScheduleGCDNM>2214</ScheduleGCDNM>
                  <LegalExceptions />
                  <Desk />
               </Flight>
            </Result>
         </GetFlightsResult>
      </GetFlightsResponse>
   </soap:Body>
</soap:Envelope>

@zenas1210
Copy link
Author

zenas1210 commented Jan 11, 2025 •
edited
Loading

I've added some debug logging locally here to see property names being handled, and saw properties like ReservationComment in the output, which isn't even close to being in GetFlights response, so maybe that has something to do with it. Haven't experienced issues with other requests though. Would love to help fix this but I'm not yet familiar with the architecture and everything.

@veewee
Copy link
Member

veewee commented Jan 11, 2025

Thanks for sharing the example. That payload should definitely not take 4G of ram.

I'll try to take a look what's going on next week when I'm back behind my computer.

@veewee
Copy link
Member

veewee commented Jan 12, 2025

Just a small heads-up:
The problem here seems recursion.

PHP Fatal error: Uncaught Error: Xdebug has detected a possible infinite loop, and aborted your script with a stack depth of '512' frames

^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"
^ "Crew"
^ "ArrayOfRosterActivity"
^ "RosterActivity"
^ "ArrayOfCrewOnBoard"
^ "CrewOnBoard"

This recursion floods your memory until it crashes (unless you have xdebug enabled)

As you can see, the types are linking to each other from within a subtype.
Currently the ObjectAccess is trying to resolve to a type that can cover every potential optional object from within the response given the XSD in your WSDL. There is no failsafe system that can detect this recursive redundant types yet.

So that's what I'm gonna look into solving for this issue.

@veewee veewee mentioned this issue Jan 12, 2025
Merged
@veewee
Copy link
Member

veewee commented Jan 12, 2025 •
edited
Loading

@zenas1210 You can test this PR out already:
#35

I still need to do some additional performance tests on big payloads - but this should be good to go.
The decoding now took about 4MB, which is doable I suppose :)

@zenas1210
Copy link
Author

woah cool I'll try this out today

@zenas1210
Copy link
Author

So far seems to work perfectly, tried out in a symfony app with a 5 megabyte response, didn't have to increase the memory limit beyond 256M. Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Load object access information lazily veewee/encoding
2 participants
@veewee @zenas1210