"Unable to get new page, browser likely crashed"

[vitaly-zdanevich]

$ docker run  -v /output:/output ghcr.io/openzim/zimit zimit --url https://flibusta.is --name flibusta --workers 100

...end of log...

5Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\",\"added\":\"2024-09-26T01:29:34.289Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.046Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:13.690Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/24\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\\/fb2\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}"]}}
{"timestamp":"2024-09-26T01:37:21.221Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":31}}
{"timestamp":"2024-09-26T01:37:21.221Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":31,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 31)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:21.300Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/comment/1599330","workerid":74}}
{"timestamp":"2024-09-26T01:37:21.721Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/g/89","workerid":31}}
{"timestamp":"2024-09-26T01:37:22.065Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":43}}
{"timestamp":"2024-09-26T01:37:22.065Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":43,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 43)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:22.566Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/b/800257/fb2","workerid":43}}
{"timestamp":"2024-09-26T01:37:22.568Z","logLevel":"warn","context":"worker","message":"Page Worker Timeout","details":{"seconds":190,"page":"https://flibusta.is/node/540115","workerid":27}}
{"timestamp":"2024-09-26T01:37:22.569Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://flibusta.is/node/540115","workerid":27}}
{"timestamp":"2024-09-26T01:37:22.599Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":27,"page":"https://flibusta.is/b/800242/download"}}
{"timestamp":"2024-09-26T01:37:22.621Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1285,"total":390998,"pending":100,"failed":2,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2024-09-26T01:35:11.172Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/node\\/266548?page=4\",\"added\":\"2024-09-26T01:29:33.872Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.511Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800256\\/read\",\"added\":\"2024-09-26T01:29:34.102Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:22.961Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307605\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.274Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\",\"added\":\"2024-09-26T01:29:34.162Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:15.069Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800242\\/read\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:57.835Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/66035\",\"added\":\"2024-09-26T01:29:34.122Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.433Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/read\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:55.107Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800252\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.863Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800255\\/mobi\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:49.053Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/read\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:46.339Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/91379\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:20.720Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\\/epub\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.063Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/mobi\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:07.267Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/88051\",\"added\":\"2024-09-26T01:29:34.303Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:14.123Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/comment\\/1599318\",\"added\":\"2024-09-26T01:29:33.884Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:55.188Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800253\\/download\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:53.489Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/180421\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:20.412Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/mobi\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.158Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307607\",\"added\":\"2024-09-26T01:29:34.127Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:20.884Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\\/mobi\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.467Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/291833\",\"added\":\"2024-09-26T01:29:34.132Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:17.401Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/epub\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:51.219Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\",\"added\":\"2024-09-26T01:29:34.302Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:49.444Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/26083\",\"added\":\"2024-09-26T01:29:34.299Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:40.386Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/read\",\"added\":\"2024-09-26T01:29:34.292Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/213\",\"added\":\"2024-09-26T01:29:34.162Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:57.835Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/62\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.863Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/32\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:42.985Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800257\\/mobi\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.847Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/106\",\"added\":\"2024-09-26T01:29:34.103Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.417Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800257\\/read\",\"added\":\"2024-09-26T01:29:34.100Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\\/read\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/275411\",\"added\":\"2024-09-26T01:29:34.295Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:14.228Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/read\",\"added\":\"2024-09-26T01:29:34.310Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:34:05.024Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/blog\\/7176?page=1\",\"added\":\"2024-09-26T01:29:32.305Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:46.355Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:11.020Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/22\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/254\",\"added\":\"2024-09-26T01:29:34.162Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.434Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/2\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.871Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800254\\/mobi\",\"added\":\"2024-09-26T01:29:34.116Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.414Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800258\\/mobi\",\"added\":\"2024-09-26T01:29:34.088Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:11.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/epub\",\"added\":\"2024-09-26T01:29:34.308Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/91131\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:11.314Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/read\",\"added\":\"2024-09-26T01:29:34.307Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.410Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/28\",\"added\":\"2024-09-26T01:29:34.086Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:22.595Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800242\\/download\",\"added\":\"2024-09-26T01:29:34.312Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.436Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/4\",\"added\":\"2024-09-26T01:29:34.281Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.963Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800257\\/epub\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:10.718Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/88732\",\"added\":\"2024-09-26T01:29:34.255Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:49.049Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/fb2\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:22.923Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/9\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.209Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/96\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:20.945Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800241\",\"added\":\"2024-09-26T01:29:34.312Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:14.267Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800242\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:10.718Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\\/read\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/fb2\",\"added\":\"2024-09-26T01:29:34.293Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:47.482Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/epub\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/mobi\",\"added\":\"2024-09-26T01:29:34.294Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.855Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307620\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:44.075Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/267209\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:20.693Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/299875\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:23.919Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/fb2\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:21.632Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/11\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:50.931Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/mobi\",\"added\":\"2024-09-26T01:29:34.299Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.209Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/70\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/197\",\"added\":\"2024-09-26T01:29:34.297Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:05.902Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/24335\",\"added\":\"2024-09-26T01:29:33.807Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.575Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/257622\",\"added\":\"2024-09-26T01:29:34.160Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.856Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800254\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.393Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800259\\/mobi\",\"added\":\"2024-09-26T01:29:34.086Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:39.879Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800258\\/fb2\",\"added\":\"2024-09-26T01:29:34.086Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:39.171Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/94812\",\"added\":\"2024-09-26T01:29:34.290Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:20.608Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/211714\",\"added\":\"2024-09-26T01:29:34.312Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.432Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/mobi\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.156Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800252\\/read\",\"added\":\"2024-09-26T01:29:34.123Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.039Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/205952\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:49.378Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800255\\/epub\",\"added\":\"2024-09-26T01:29:34.103Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:11.987Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/fb2\",\"added\":\"2024-09-26T01:29:34.308Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.527Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/fb2\",\"added\":\"2024-09-26T01:29:34.310Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:23.483Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:11.174Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/node\\/266548?page=5\",\"added\":\"2024-09-26T01:29:33.872Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.447Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/epub\",\"added\":\"2024-09-26T01:29:34.293Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.065Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/93926\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:29.651Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800260\\/epub\",\"added\":\"2024-09-26T01:29:34.082Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307604\",\"added\":\"2024-09-26T01:29:34.296Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:23.539Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/epub\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:05.762Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800250\",\"added\":\"2024-09-26T01:29:34.239Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:44.014Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/201\",\"added\":\"2024-09-26T01:29:34.297Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.867Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800254\\/epub\",\"added\":\"2024-09-26T01:29:34.116Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:31.395Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800260\\/mobi\",\"added\":\"2024-09-26T01:29:34.082Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:14.233Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/298077\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:10.718Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.607Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/61\",\"added\":\"2024-09-26T01:29:34.200Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.608Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800255\\/read\",\"added\":\"2024-09-26T01:29:34.103Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.481Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800256\\/epub\",\"added\":\"2024-09-26T01:29:34.102Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.493Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800256\\/mobi\",\"added\":\"2024-09-26T01:29:34.102Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.985Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\",\"added\":\"2024-09-26T01:29:34.289Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.046Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:13.690Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/24\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\\/fb2\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}"]}}
{"timestamp":"2024-09-26T01:37:22.626Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":40}}
{"timestamp":"2024-09-26T01:37:22.627Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":40,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 40)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:22.972Z","logLevel":"warn","context":"worker","message":"Page Worker Timeout","details":{"seconds":190,"page":"https://flibusta.is/blog/7176?page=1","workerid":38}}
{"timestamp":"2024-09-26T01:37:22.973Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://flibusta.is/blog/7176?page=1","workerid":38}}
{"timestamp":"2024-09-26T01:37:22.997Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":38,"page":"https://flibusta.is/a/291830"}}
{"timestamp":"2024-09-26T01:37:22.999Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1286,"total":391652,"pending":100,"failed":2,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2024-09-26T01:35:11.172Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/node\\/266548?page=4\",\"added\":\"2024-09-26T01:29:33.872Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.511Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800256\\/read\",\"added\":\"2024-09-26T01:29:34.102Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:22.961Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307605\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.274Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\",\"added\":\"2024-09-26T01:29:34.162Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:15.069Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800242\\/read\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:57.835Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/66035\",\"added\":\"2024-09-26T01:29:34.122Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.433Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/read\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:55.107Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800252\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.863Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800255\\/mobi\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:49.053Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/read\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:46.339Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/91379\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:20.720Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\\/epub\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.063Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/mobi\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:07.267Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/88051\",\"added\":\"2024-09-26T01:29:34.303Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:14.123Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/comment\\/1599318\",\"added\":\"2024-09-26T01:29:33.884Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:55.188Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800253\\/download\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:53.489Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/180421\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:20.412Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/mobi\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.158Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307607\",\"added\":\"2024-09-26T01:29:34.127Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:20.884Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\\/mobi\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.467Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/291833\",\"added\":\"2024-09-26T01:29:34.132Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:17.401Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/epub\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:51.219Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\",\"added\":\"2024-09-26T01:29:34.302Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:49.444Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/26083\",\"added\":\"2024-09-26T01:29:34.299Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:40.386Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/read\",\"added\":\"2024-09-26T01:29:34.292Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/213\",\"added\":\"2024-09-26T01:29:34.162Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:57.835Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/62\",\"added\":\"2024-09-26T01:29:34.121Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.863Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/32\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:42.985Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800257\\/mobi\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.847Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/106\",\"added\":\"2024-09-26T01:29:34.103Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.417Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800257\\/read\",\"added\":\"2024-09-26T01:29:34.100Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\\/read\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/275411\",\"added\":\"2024-09-26T01:29:34.295Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:14.228Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/read\",\"added\":\"2024-09-26T01:29:34.310Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:46.355Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:11.020Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/22\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/254\",\"added\":\"2024-09-26T01:29:34.162Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.434Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/2\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.871Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800254\\/mobi\",\"added\":\"2024-09-26T01:29:34.116Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.414Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800258\\/mobi\",\"added\":\"2024-09-26T01:29:34.088Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:11.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/epub\",\"added\":\"2024-09-26T01:29:34.308Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/91131\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:11.314Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/read\",\"added\":\"2024-09-26T01:29:34.307Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.410Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/28\",\"added\":\"2024-09-26T01:29:34.086Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:22.595Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800242\\/download\",\"added\":\"2024-09-26T01:29:34.312Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.436Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/4\",\"added\":\"2024-09-26T01:29:34.281Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.963Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800257\\/epub\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:10.718Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/88732\",\"added\":\"2024-09-26T01:29:34.255Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:49.049Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/fb2\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:22.923Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/9\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.209Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/96\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:20.945Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800241\",\"added\":\"2024-09-26T01:29:34.312Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:14.267Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800242\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:10.718Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\\/read\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/fb2\",\"added\":\"2024-09-26T01:29:34.293Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:47.482Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/epub\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/mobi\",\"added\":\"2024-09-26T01:29:34.294Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.855Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307620\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:22.997Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/291830\",\"added\":\"2024-09-26T01:29:34.313Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:44.075Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/267209\",\"added\":\"2024-09-26T01:29:34.298Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:20.693Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/299875\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:23.919Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/fb2\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:21.632Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/11\",\"added\":\"2024-09-26T01:29:34.258Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:50.931Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800246\\/mobi\",\"added\":\"2024-09-26T01:29:34.299Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.209Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/70\",\"added\":\"2024-09-26T01:29:34.101Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/197\",\"added\":\"2024-09-26T01:29:34.297Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:05.902Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/24335\",\"added\":\"2024-09-26T01:29:33.807Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.575Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/257622\",\"added\":\"2024-09-26T01:29:34.160Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.856Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800254\",\"added\":\"2024-09-26T01:29:34.115Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:40.393Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800259\\/mobi\",\"added\":\"2024-09-26T01:29:34.086Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:39.879Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800258\\/fb2\",\"added\":\"2024-09-26T01:29:34.086Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:39.171Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/94812\",\"added\":\"2024-09-26T01:29:34.290Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:20.608Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/211714\",\"added\":\"2024-09-26T01:29:34.312Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.432Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/mobi\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:59.156Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800252\\/read\",\"added\":\"2024-09-26T01:29:34.123Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.039Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/205952\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:49.378Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800255\\/epub\",\"added\":\"2024-09-26T01:29:34.103Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:11.987Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800245\\/fb2\",\"added\":\"2024-09-26T01:29:34.308Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.527Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\\/fb2\",\"added\":\"2024-09-26T01:29:34.310Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:23.483Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:11.174Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/node\\/266548?page=5\",\"added\":\"2024-09-26T01:29:33.872Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.447Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\\/epub\",\"added\":\"2024-09-26T01:29:34.293Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.065Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/s\\/93926\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:29.651Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800260\\/epub\",\"added\":\"2024-09-26T01:29:34.082Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:43.946Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/307604\",\"added\":\"2024-09-26T01:29:34.296Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:23.539Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800248\\/epub\",\"added\":\"2024-09-26T01:29:34.259Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:05.762Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800250\",\"added\":\"2024-09-26T01:29:34.239Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:44.014Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/201\",\"added\":\"2024-09-26T01:29:34.297Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.867Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800254\\/epub\",\"added\":\"2024-09-26T01:29:34.116Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:31.395Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800260\\/mobi\",\"added\":\"2024-09-26T01:29:34.082Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:14.233Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/a\\/298077\",\"added\":\"2024-09-26T01:29:34.311Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:10.718Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800249\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.607Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/61\",\"added\":\"2024-09-26T01:29:34.200Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:51.608Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800255\\/read\",\"added\":\"2024-09-26T01:29:34.103Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.481Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800256\\/epub\",\"added\":\"2024-09-26T01:29:34.102Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:35:47.493Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800256\\/mobi\",\"added\":\"2024-09-26T01:29:34.102Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:38.985Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800247\",\"added\":\"2024-09-26T01:29:34.289Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:37:12.046Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800244\",\"added\":\"2024-09-26T01:29:34.309Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:13.690Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/g\\/24\",\"added\":\"2024-09-26T01:29:34.257Z\",\"depth\":2}","{\"seedId\":0,\"started\":\"2024-09-26T01:36:03.275Z\",\"extraHops\":0,\"url\":\"https:\\/\\/flibusta.is\\/b\\/800251\\/fb2\",\"added\":\"2024-09-26T01:29:34.163Z\",\"depth\":2}"]}}
{"timestamp":"2024-09-26T01:37:23.128Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/g/44","workerid":40}}
{"timestamp":"2024-09-26T01:37:23.812Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800259/mobi","workerid":90}}
{"timestamp":"2024-09-26T01:37:23.908Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":21}}
{"timestamp":"2024-09-26T01:37:23.908Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":21,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 21)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:23.910Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":69}}
{"timestamp":"2024-09-26T01:37:23.910Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":69,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 69)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:24.268Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/s/93713","workerid":82}}
{"timestamp":"2024-09-26T01:37:24.268Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800253","workerid":50}}
{"timestamp":"2024-09-26T01:37:24.349Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":55}}
{"timestamp":"2024-09-26T01:37:24.349Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":55,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 55)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)"}}
{"timestamp":"2024-09-26T01:37:24.349Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":41}}
{"timestamp":"2024-09-26T01:37:24.349Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":41,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 41)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:24.408Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/comment/1599346","workerid":21}}
{"timestamp":"2024-09-26T01:37:24.411Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/comment/1599559","workerid":69}}
{"timestamp":"2024-09-26T01:37:24.849Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/s/94814","workerid":55}}
{"timestamp":"2024-09-26T01:37:24.850Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/g/99","workerid":41}}
{"timestamp":"2024-09-26T01:37:25.847Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://flibusta.is/s/93926","workerid":26}}
{"timestamp":"2024-09-26T01:37:27.405Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800258/epub","workerid":30}}
{"timestamp":"2024-09-26T01:37:27.543Z","logLevel":"warn","context":"general","message":"Cloudflare check timed out","details":{"seconds":5,"page":"https://flibusta.is/a/291833","workerid":48}}
{"timestamp":"2024-09-26T01:37:27.789Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://flibusta.is/b/800245/fb2","workerid":98}}
{"timestamp":"2024-09-26T01:37:28.476Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800256/mobi","workerid":3}}
{"timestamp":"2024-09-26T01:37:28.770Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800260/epub","workerid":34}}
{"timestamp":"2024-09-26T01:37:28.771Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800257/epub","workerid":91}}
{"timestamp":"2024-09-26T01:37:28.833Z","logLevel":"error","context":"general","message":"Page Load Failed, skipping page","details":{"msg":"Navigation timeout of 90000 ms exceeded","loadState":0,"page":"https://flibusta.is/node/266548?page=4","workerid":53}}
{"timestamp":"2024-09-26T01:37:29.603Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/b/800258/mobi","workerid":35}}
{"timestamp":"2024-09-26T01:37:30.412Z","logLevel":"error","context":"worker","message":"Page Close Timed Out","details":{"seconds":10,"page":"https://flibusta.is/a/291834","workerid":96}}
{"timestamp":"2024-09-26T01:37:31.100Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":37}}
{"timestamp":"2024-09-26T01:37:31.101Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":37,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 37)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:31.600Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/a/307621","workerid":37}}
{"timestamp":"2024-09-26T01:37:32.719Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://flibusta.is/g/32","page":"https://flibusta.is/g/32","workerid":61}}
{"timestamp":"2024-09-26T01:37:32.719Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://flibusta.is/g/32","workerid":61}}
{"timestamp":"2024-09-26T01:37:33.720Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://flibusta.is/g/32","workerid":61}}
{"timestamp":"2024-09-26T01:37:33.945Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":93}}
{"timestamp":"2024-09-26T01:37:33.945Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":93,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at listOnTimeout (node:internal/timers:545:9)\n    at process.processTimers (node:internal/timers:519:7)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 93)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)"}}
{"timestamp":"2024-09-26T01:37:33.946Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":68}}
{"timestamp":"2024-09-26T01:37:33.946Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":68,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 68)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:34.038Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":44}}
{"timestamp":"2024-09-26T01:37:34.039Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":44,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 44)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:34.445Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/b/800255","workerid":93}}
{"timestamp":"2024-09-26T01:37:34.445Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/a/110865","workerid":68}}
{"timestamp":"2024-09-26T01:37:34.538Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/b/800259/fb2","workerid":44}}
{"timestamp":"2024-09-26T01:37:34.560Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":28}}
{"timestamp":"2024-09-26T01:37:34.561Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":28,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 28)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:35.070Z","logLevel":"warn","context":"worker","message":"Retrying getting new page","details":{"page":"https://flibusta.is/b/800257","workerid":28}}
{"timestamp":"2024-09-26T01:37:36.349Z","logLevel":"error","context":"worker","message":"New Window Timed Out","details":{"seconds":20,"workerid":59}}
{"timestamp":"2024-09-26T01:37:36.349Z","logLevel":"warn","context":"worker","message":"Error getting new page","details":{"workerid":59,"type":"exception","message":"timed out","stack":"Error: timed out\n    at PageWorker.initPage (file:///app/dist/util/worker.js:93:27)\n    at async PageWorker.runLoop (file:///app/dist/util/worker.js:226:30)\n    at async PageWorker.run (file:///app/dist/util/worker.js:199:13)\n    at async Promise.allSettled (index 59)\n    at async runWorkers (file:///app/dist/util/worker.js:276:5)\n    at async Crawler.crawl (file:///app/dist/crawler.js:913:9)\n    at async Crawler.run (file:///app/dist/crawler.js:354:13)\n    at async file:///app/dist/main.js:49:1"}}
{"timestamp":"2024-09-26T01:37:36.349Z","logLevel":"fatal","context":"worker","message":"Unable to get new page, browser likely crashed. Quitting","details":{"page":"https://flibusta.is/node/266548?page=3","workerid":59}}
Traceback (most recent call last):
  File "/usr/bin/zimit", line 8, in <module>
    sys.exit(zimit.zimit())
             ^^^^^^^^^^^^^
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 688, in zimit
    run(sys.argv[1:])
  File "/app/zimit/lib/python3.12/site-packages/zimit/zimit.py", line 574, in run
    raise subprocess.CalledProcessError(crawl.returncode, cmd_args)
subprocess.CalledProcessError: Command '['crawl', '--failOnFailedSeed', '--workers', '100', '--waitUntil', 'load', '--depth', '-1', '--timeout', '90', '--behaviors', 'autoplay,autofetch,siteSpecific', '--behaviorTimeout', '90', '--diskUtilization', '90', '--url', 'https://flibusta.is', '--userAgentSuffix', '+Zimit', '--mobileDevice', 'Pixel 2', '--cwd', '/output/.tmp1_vp3h0c']' returned non-zero exit status 17.
[zimit::2024-09-26 01:37:36,618] INFO:
[zimit::2024-09-26 01:37:36,619] INFO:----------
[zimit::2024-09-26 01:37:36,619] INFO:Cleanup, removing temp dir: /output/.tmp1_vp3h0c```

[vitaly-zdanevich]

The same with 50 workers.

[benoit74]

This is a puppeteer issue, probably hard to fix.

Quite close to #376

@ikreymer is this something currently under the radar of browsertrix team with an issue we can link to?

[ikreymer]

This may or may not be related to #376, since this doesn't crash node, but seems like browser may be crashing eventually. 100 or 50 works is quite a lot, unless you have an incredible powerful machine, the RAM/CPU requirements for running 50 browser windows in parallel is likely not going to be enough. I would try with 2 or 4 workers first.

[vitaly-zdanevich]

Running on old laptop: 16 GB of DDR3 RAM.

For the last 2 days running with 20 workers - no errors.

[ikreymer]

Running on old laptop: 16 GB of DDR3 RAM.

For the last 2 days running with 20 workers - no errors.

Glad that it's working! For a most static site, the memory usage is much lower. If you run into issues, would try with less workers.