How to start the schedulers in esmerald.

[harshalizode]

Hey,
I would like to run certain schedulers, but I'm not sure if esmearld initiates scheduling automatically if enable_scheduler=True is set in the app instance.

I have set up the scheduler according to what the documentation mentions. However, when the application launches, the message appears as Adding task tentatively. It will be properly scheduled when the scheduler starts.. When the scheduler launches, it will be correctly scheduled but not executed.

[tarsil]

@harshalizode did you map the schedule tasks into the proper configuration? Something like this:

from esmerald import Esmerald

app = Esmerald(
    routes=[...],
    enable_scheduler=True,
    scheduler_tasks={
        "collect_market_data": "accounts.tasks",
        "send_email_newsletter": "accounts.tasks",
    },
)

The collect_market_data and send_email_newsletter is the nameof the task that is located inside an accounts.tasks file. In your case would be something different, of course

[harshalizode]

I have configured as:

from esmerald import Esmerald

app = Esmerald(
    routes=[...],
    scheduler_class=AsyncIOScheduler,
    enable_scheduler=True,
    scheduler_tasks={
        "scheduler_demo": "proj.apps.common.tasks"
    },
    scheduler_configurations=scheduler_configurations,
)

My scheduler as:

from asyncz.contrib.esmerald.decorator import scheduler
from asyncz.triggers import IntervalTrigger
from esmerald.conf import settings

logger = settings.logger
count = 0


@scheduler(
    name="print_no_of_running_count",
    trigger=IntervalTrigger(seconds=10),
    max_instances=3,
    store="mongo",
    executor="default",
)
def scheduler_demo():
    global count
    count += 1
    print(f"Scheduler running count {count}")

This is where I have a doubt: the application logs a message Adding task tentatively. It will be properly scheduled when the scheduler starts when I launch it. My scheduler doesn't print anything on the console.

[tarsil]

@harshalizode Ok, I'm adding an extra internal validation and I will do a quick release soon (was planned anyway) and I will push an example of a project using the scheduler.

I tested now and it works as intended but you can see how it was done. Pretty much like the docs but maybe you missed something out or it was not 100% clear

See the example running below

Give me a few minutes and I will come back to you.

[tarsil]

@harshalizode A new release happened and there is a project example https://github.com/dymmond/scheduler-example where you can see the above being replicated.

[harshalizode]

Thank you for your quick response.

[tarsil]

Is it good now?

[tarsil]

@harshalizode is it ok now? Btw the mongoz will be addressed this week. Sorry but we have been busy lately with Esmerald, Lilya and Edgy

[tarsil]

@harshalizode Esmerald uses what Asyncz provides in the contrib. Automatically starts the scheduler for you without you worrying too much about it by design. The AsyncIOScheduler is the default one if none is provided.

Esmerald calls the EsmeraldScheduler provided by Asyncz and automatically, Asyncz boots up like this

https://github.com/dymmond/asyncz/blob/main/asyncz/contrib/esmerald/scheduler.py#L73

That will automatically do this: https://github.com/dymmond/asyncz/blob/main/asyncz/contrib/esmerald/scheduler.py#L92 for you

You can create your own AsyncIOScheduler if you want and pass it to scheduler_class parameter in Esmerald but if you don't do, Esmerald will tell Asyncz to use the AsyncIOScheduler default and that EsmeraldScheduler class does the automatic heavy lifting for you.

[harshalizode]

@tarsil is the AsyncIOScheduler is default one?
If we not provide the scheduler_class via creating app instance then it raise the exception for it.

[harshalizode]

@tarsil I have go through the code that you have provided as link.
The message is showing as the Adding task tentatively. It will be properly scheduled when the scheduler starts.
Means it is waiting for scheduler to start. I have also debug the startup(start_scheduler) and shutdown(stop_scheduler) script in the register_events(), but it never runs by application starts and end

[tarsil]

II

@tarsil is the AsyncIOScheduler is default one? If we not provide the scheduler_class via creating app instance then it raise the exception for it.

Please install the latest version of Esmerald. That was addressed yesterday and now it's even cleaner.

Regarding your other answer. Did you do like the example provided? Because literally that starts the scheduler but maybe you need to install the latest version of Esmerald to address the other situation. Install the version 3.2.2 and then give it a try.

We have updated the internals to make sure it does a cleaner validation upon instantiation of the scheduler

[harshalizode]

Okay, I'll install the recent version of Esmerald 3.2.2 and notify you about the scheduler.
Thank you.

[harshalizode]

Hi @tarsil , The issue was the same, the task was added to the scheduler but the scheduler was not started; however, I no longer need to give the scheduler_class via app instance.
You have created the startup script to start the scheduler and the shoutdown script to stop it, but they never run throughout the program when it starts or stops.

[tarsil]

What do you mean? This is like the example. Basically like that you enable the scheduler. In the example, that triggers the scheduler and bootsup

[tarsil]

That do_startup, what is it? The scheduler? You don't need that . Esmerald handles that for you when you pass the tasks and the enable_scheduler

[tarsil]

Yes, I found the root cause of it. If we add the startup script as:

app = Esmerald(
        routes=[Include(namespace="scheduler_example.urls")],
        enable_scheduler=True,
        scheduler_tasks={
            "collect_market_data": "scheduler_example.apps.accounts.tasks",
            "send_newsletter": "scheduler_example.apps.accounts.tasks",
            "scheduler_demo": "scheduler_example.apps.accounts.tasks",
        },
        on_startup=[do_startup]
    )
    return app

Then it won't able to execute the scheduler's script, because we are overriding this via app instantiation.

Ohh wait. If you remove the "do_startup". Does it work? Because if it does, then I think I might know what the problem might be and I need to upgrade the asyncz

[tarsil]

@harshalizode I was able to replicate the issue based on my question. On it.

[harshalizode]

Yes

Yes, I found the root cause of it. If we add the startup script as:

app = Esmerald(
        routes=[Include(namespace="scheduler_example.urls")],
        enable_scheduler=True,
        scheduler_tasks={
            "collect_market_data": "scheduler_example.apps.accounts.tasks",
            "send_newsletter": "scheduler_example.apps.accounts.tasks",
            "scheduler_demo": "scheduler_example.apps.accounts.tasks",
        },
        on_startup=[do_startup]
    )
    return app

Then it won't able to execute the scheduler's script, because we are overriding this via app instantiation.

Ohh wait. If you remove the "do_startup". Does it work? Because if it does, then I think I might know what the problem might be and I need to upgrade the asyncz

Yes absolutely, If I remove the on_startup argument, then its work.
do_startup() is normal sync function that print "Startup Completed.."

[tarsil]

@harshalizode there was a reimplementation of Asyncz for this case since Esmerald was updated to Lilya in the past and that reflects also in the initialisation of Esmerald in the new version 3.2.3.

If you install the latest version 3.2.3 of Esmerald, you issue should now be fixed. Thank you for reporting this.

Let me know once you do it.

[tarsil]

Also, feel free to join our discord https://discord.gg/haFGp3CK

We usually talk a lot there and we have people that are able to also assist you.

[harshalizode]

While running the application getting the issue as:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/harshaliz/.env/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 78, in subprocess_started
    target(sockets=sockets)
  File "/home/harshaliz/.env/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/harshaliz/.env/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve
    await self._serve(sockets)
  File "/home/harshaliz/.env/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve
    config.load()
  File "/home/harshaliz/.env/lib/python3.10/site-packages/uvicorn/config.py", line 433, in load
    self.loaded_app = import_from_string(self.app)
  File "/home/harshaliz/.env/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/main.py", line 98, in <module>
    app = get_application()
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/main.py", line 52, in get_application
    app = Esmerald(
  File "/home/harshaliz/.env/lib/python3.10/site-packages/esmerald/applications.py", line 1630, in __init__
    self.activate_scheduler()
  File "/home/harshaliz/.env/lib/python3.10/site-packages/esmerald/applications.py", line 1767, in activate_scheduler
    self.scheduler = EsmeraldScheduler(
  File "/home/harshaliz/.env/lib/python3.10/site-packages/asyncz/contrib/esmerald/scheduler.py", line 73, in __init__
    self.register_events(app=self.app)
  File "/home/harshaliz/.env/lib/python3.10/site-packages/asyncz/contrib/esmerald/scheduler.py", line 97, in register_events
    @app.on_event("startup")
  File "/home/harshaliz/.env/lib/python3.10/site-packages/esmerald/applications.py", line 2622, in on_event
    return self.router.on_event(event_type)
AttributeError: 'Esmerald' object has no attribute 'router'. Did you mean: 'route'?

[tarsil]

@harshalizode sorry but for some reason the latest asyncz is not installed in your app. So when you install with the latest should be like you did before esmerald[schedulers] or install the asyncz>=0.7.0 as well.

This was a change on both sides. Esmerald by default does not install asyncz unless you say like you did with esmerald[schedulers]

[tarsil]

Because you have an older version of asyncz, its still trying to execute that.

[tarsil]

@harshalizode In simple terms. What you reported was actually coming from asyncz directly and that needed updating. Esmerald to reflect that change, updated the startup process for it.

Because you have installed already both you need to

pip install esmerald==3.2.3 asyncz==0.7.0

This should do it

[harshalizode]

Yes, Its working fine now.
Thank you for your consideration of it.

[harshalizode]

There is one more issue, I want to use the mongoDB as scheduler store. But adding the configuration of scheduler it raising exceptions.
To configure the scheduler I am doing same as mentioned in the Asyncz documentation.

Issue as:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/proj.env/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 78, in subprocess_started
    target(sockets=sockets)
  File "/home/proj.env/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/proj.env/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve
    await self._serve(sockets)
  File "/home/proj.env/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve
    config.load()
  File "/home/proj.env/lib/python3.10/site-packages/uvicorn/config.py", line 433, in load
    self.loaded_app = import_from_string(self.app)
  File "/home/proj.env/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/main.py", line 19, in <module>
    from playmity.startup_scripts import load_permissions, \
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/startup_scripts.py", line 17, in <module>
    from common.decorator import require_application_models
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/common/decorator.py", line 17, in <module>
    logger = settings.logger
  File "/home/proj.env/lib/python3.10/site-packages/esmerald/utils/functional.py", line 12, in inner
    self._setup()
  File "/home/proj.env/lib/python3.10/site-packages/esmerald/conf/__init__.py", line 40, in _setup
    settings: Type["EsmeraldAPISettings"] = reload_settings()
  File "/home/proj.env/lib/python3.10/site-packages/esmerald/conf/__init__.py", line 23, in reload_settings
    settings: Type["EsmeraldAPISettings"] = import_string(settings_module)
  File "/home/proj.env/lib/python3.10/site-packages/lilya/_internal/_module_loading.py", line 26, in import_string
    module = import_module(module_path)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/configs/development/__init__.py", line 5, in <module>
    from .settings import DevelopmentAppSettings
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/configs/development/settings.py", line 54, in <module>
    class DevelopmentAppSettings(AppSettings):
  File "/home/harshaliz/Desktop/Projects/Playmity/trunk/playmity_be/playmity/configs/development/settings.py", line 173, in DevelopmentAppSettings
    scheduler: AsyncIOScheduler = AsyncIOScheduler(
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/asyncio.py", line 24, in __init__
    super().__init__(**kwargs)
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/base.py", line 103, in __init__
    self.setup(self.global_config, **kwargs)
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/base.py", line 152, in setup
    self._setup(config)
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/asyncio.py", line 40, in _setup
    super()._setup(config)
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/base.py", line 721, in _setup
    self.add_store(value, alias)
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/base.py", line 320, in add_store
    self.wakeup()
  File "/home/proj.env/lib/python3.10/site-packages/asyncz/schedulers/utils.py", line 13, in wrapper
    self.event_loop.call_soon_threadsafe(wrapped)
AttributeError: 'NoneType' object has no attribute 'call_soon_threadsafe'

[tarsil]

Yes, Its working fine now. Thank you for your consideration of it.

Not a problem. Like I mentioned, it was from asyncz. Esmerald was always ok. Regarding your other issue, lets move the conversation to asyncz? This way we can focus the efforts there? This way we can address this issue. Maybe a configuration is not being parsed properly or something and we can quickly address and you can simply update the asyncz on your side.

Btw, besides these small issues, I hope Esmerald is ok for you.

[tarsil]

Open the issue there and do me a favour, paste the configuration you are passing, please

[harshalizode]

Okay, I will add into the Asyncz issues section.