Dumper::asJsonObjectsMap() produces out of memory error

[olegbaturin]

What steps will reproduce the problem?

run this app https://github.com/olegbaturin/yii3-app-api

Ошибка в алгоритме и довольно нетривиальная, опишу на русском.

yii-debug/src/Dumper.php

Line 62 in 7e1c705

private function buildObjectsCache(mixed $variable, int $depth, int $level = 0): void

Метод buildObjectsCache() проходит на заданную глубину $depth и строит плоский массив из всей вложенной структуры объектов, который сохраняет в $this->objects. Алгоритм также отслеживает циклические зависимости вида parent->child->parent.
Таким образом оказывается, что объект с заданной максимальной глубины вложенности $depth оказывается на первом уровне.

yii-debug/src/Dumper.php

Line 59 in 7e1c705

return $this->asJsonInternal($this->objects, $prettyPrint, $depth, 1, true);

Далее происходит этот вызов и каждый элемент из $this->objects в методе asJsonInternal() еще раз парсится на глубину $depth. Получается, что элемент из массива $this->objects с глубины $depth будет парситься повторно на эту же глубину $depth, т.е. в итоге 2* $depth.

yii-debug/src/Dumper.php

Line 162 in 7e1c705

if ($objectCollapseLevel < $level && array_key_exists($objectDescription, $this->objects)) {

Ошибка возникает из-за того, что в этой проверке предполагается, что все объекты уже есть в кэшэ, т.е. в массиве $this->objects. Это не так, потому что $this->objects содержит объекты с глубины только $depth, а парсинг делается на глубину 2* $depth. Также метод dumpNestedInternal() не отслеживает цикличекские зависимости parent->child->parent.

Если конфигурация приложения содержит циклические зависимости parent->child->parent , то происходит цикличкский дамп до глубины 2* $depth без учёта вложенных дублей.

Additional info

Q	A
Version	dev
PHP version	any
Operating system	any

[xepozz]

Why there was memory leaks

• The Dumper goes through the passed object and collects all possible object references being limited to passed "depth" or 50 as default
• After the first step is done the Dumper starts unpack/print/dump the object
• Default dump "depth" for FileStorage is 30

So what may go wrong?

Example

1. Here we have an recursive structure:

• node1.prop refers to node2
• node2.prop refers to node3
• etc

2. Create a var and pass top node to the dumper, set a custom limit to see the result in a few lines. I'd set it to 4:

Dumper::create($var)->asJsonObjectsMap(4)

3. Run the code and see the output:

{
  "stdClass#2488": {
    "public $objects": [],
    "public $id": "lvl1",
    "public $lvl2": "object@stdClass#1067" <------- refers to next object
  },
  "stdClass#1067": {
    "public $id": "lvl2",
    "public $lvl3": "object@stdClass#2275" <------- refers to next object
  },
  "stdClass#2275": {
    "public $id": "lvl3",
    "public $lvl4": "object@stdClass#995" <------- refers to next object
  },
  "stdClass#995": {
    "public $id": "lvl4",
    "public $lvl5": { <----------- here must be object reference, but which?
      "public $id": "lvl5",
      "public $lvl6": { 
        "public $id": "lvl6",
        "public $lvl7": "stdClass#2491 (...)"
      }
    }
  }
}

Gotcha. Seems like the last object does not collapse like the rest do. Why?

Codebase

I've checked the codebase for object@ usage and here the one condition to collapse the object:

if ($objectCollapseLevel < $level && array_key_exists($objectDescription, $this->objects)) {
    $output = 'object@' . $objectDescription;
    break;
}

What do they mean:

1. $objectCollapseLevel < $level – if the recursion level (depth) is greater than a $objectCollapseLevel. What is this?
2. array_key_exists($objectDescription, $this->objects) – we know what's the object, scanned it in the buildObjectsCache method previously

$objectCollapseLevel – is an static value with specific values only:

So that condition fails if we don't know the object, that's why parsing goes through the object "in depth" and we can see the following example for 100 recursive elements and dumping depth limited to 50:

All scanned elements, limited by 50:

And dumped in depth the last object, also limited by 50:

So everything works great? No really, going deeper...

Example 2

Imagine the same recursive structure, but with one change: we have two properties that's refers to the same "next" element:

• node1.prop1 and node1.prop2 refers to node2
• node2.prop1 and node2.prop2 refers to node3
• etc

Let's dump it:

{
  "stdClass#2344": {
    "public $objects": [],
    "public $id": "lvl1",
    "public $prop1": "object@stdClass#773",
    "public $prop2": "object@stdClass#773"
  },
  "stdClass#773": {
    "public $id": "lvl2",
    "public $prop1": "object@stdClass#2512",
    "public $prop2": "object@stdClass#2512"
  },
  "stdClass#2512": {
    "public $id": "lvl3",
    "public $prop1": "object@stdClass#2541",
    "public $prop2": "object@stdClass#2541"
  },
  "stdClass#2541": {
    "public $id": "lvl4",
    "public $prop1": { <------- refers to the object lvl5
      "public $id": "lvl5",
      "public $prop1": { <------- refers to the object lvl6
        "public $id": "lvl6",
        "public $prop1": "stdClass#2347 (...)",
        "public $prop2": "stdClass#2347 (...)"
      },
      "public $prop2": { <------- refers to the object lvl6
        "public $id": "lvl6",
        "public $prop1": "stdClass#2347 (...)",
        "public $prop2": "stdClass#2347 (...)"
      }
    },
    "public $prop2": { <------- refers to the object lvl5
      "public $id": "lvl5",
      "public $prop1": { <------- refers to the object lvl6
        "public $id": "lvl6",
        "public $prop1": "stdClass#2347 (...)",
        "public $prop2": "stdClass#2347 (...)"
      },
      "public $prop2": { <------- refers to the object lvl6
        "public $id": "lvl6",
        "public $prop1": "stdClass#2347 (...)",
        "public $prop2": "stdClass#2347 (...)"
      }
    }
  }
}

Gotcha! Again.

So here we can see how the output is doubled and every next level generates a huge flood:

• 2 dumps of lvl5
• 4 dumps of lvl6
• 2**N dumps of next level

Solution

I think it'd be great to force objects to be collapsed even if we haven't reached them while "warming up" the cache.
So the last object must collapse into:

{
"stdClass#2541": {
    "public $id": "lvl4",
    "public $prop1": { <------- refers to the object lvl5
      "public $id": "lvl5",
      "public $prop1": { <------- refers to the object lvl6
        "public $id": "lvl6",
        "public $prop1": "stdClass#2347 (...)",
        "public $prop2": "stdClass#2347 (...)"
      },
      "public $prop2": "object@ID_OF_LVL6_OBJECT"
    },
    "public $prop2": "object@ID_OF_LVL5_OBJECT"
  }
}

Looks great, isn't it?
But if we replace dumps with the representative links, we can't reach the object 'cause it was inlined just right now. Need to move it to the top. And collapse the value back top to the object dump start.

Need time to think about the solution.

[olegbaturin]

I think it'd be great to force objects to be collapsed even if we haven't reached them while "warming up" the cache.

If an object is not in the cache, it means that the object is deeper in the structure than the depth limit. The solution is to show for those objects only their ID: "stdClass#2347 (...)".

[xepozz]

This is not var-dumper package that's used to dump structures to see what they mean, it's debug package. It's better to save as much as possible data to analyze it later.

[samdark]

@olegbaturin try master on the case when it failed please.

[vjik]

Done by #287