diff --git a/README.md b/README.md index ba73202..c317085 100644 --- a/README.md +++ b/README.md @@ -49,14 +49,11 @@ Run the following command and follow the instructions To utilize SimNotify you will need to create a notify channel so that your phone (or device) can be linked to where the notifications will be sent. Go to https://notify.run/ and click `create a channel` Scan the QR code and subscribe/enable notifications (you may need to check your system settings to ensure these are turned on) -Copy the link generated, and set the config files on your machine to this link by running: -```notify-run configure ``` - -Now you should be able to run the python script: +Copy the link generated, and you should be able to run the python script: ``` from SimLogger import SimNotify endpoint = "" -SimNotify.sendNotification("Hello World", endpoint=endpoint) +SimNotify.sendNotification("Hello World", endpoint) ``` ## Development diff --git a/docs/source/FigLogger.rst b/docs/source/FigLogger.rst new file mode 100644 index 0000000..69651f3 --- /dev/null +++ b/docs/source/FigLogger.rst @@ -0,0 +1,59 @@ +SimLogger.FigLogger module +========================== + +The FigLogger is designed to solve the problem of not being able to easily +view detailed graphs through a command line interface with remote computing +systems. To solve this, an integration between the SimLogger for data archiving +and kachery were combined to give remote access to data through a web interface. + +Installation +------------ + +After following the initial installation steps for the SimLogger package. +Ensure the machine is setup to access the kachery cloud + +``kachery-cloud-init`` + +and follow the steps to setup the connection. + +With that you can now generate your own graphs, and then remotely view them. +The graphs are also saved to file using the SimLogger so you can always +simply load up the pickled version of the graph and show it again on a local +machine if desired. + +The simplest method is to let FigLogger generate the plot for you. This is +great for when you want a simple scatter or line plot without much specifics. + +.. code-block:: python + from SimLogger import FigLogger + + simTag = "testSim" + objTag = "samplePlot" + + epoch = [0,1,2,3,4] + loss = [3.0, 2.7, 2.6, 2.5, 2.45] + + plotType = "line" + title = "Sample Plot" + labels={"x":"Epoch","y":"Loss"} + + FigLogger.createPlot(simTag, + objTag, + epoch, + loss, + plotType=plotType, + title=title, + labels=labels) + + +Which will automatically generate a line plot with the data given, and will +give a link to the graph in the log file. + +SimLogger.FigLogger functions +----------------------------- + +.. automodule:: SimLogger.FigLogger + :members: + :undoc-members: + :show-inheritance: + diff --git a/docs/source/SimLogger.rst b/docs/source/SimLogger.rst index 436f874..d644609 100644 --- a/docs/source/SimLogger.rst +++ b/docs/source/SimLogger.rst @@ -1,26 +1,63 @@ -SimLogger package -================= - SimLogger.SimLogger module --------------------------- +========================== -.. automodule:: SimLogger.SimLogger - :members: - :undoc-members: - :show-inheritance: +The SimLogger module is the fundamental piece of the SimLogger package. -SimLogger.FigLogger module --------------------------- +The SimLogger functions on the key principal that everything in a simulation +can be uniquely identified with the following: +1. simTag - uniquely identifies a particular simulation configuration. + This is changed for every single test run. Everytime the parameters + are changed (and you re-run your simulation) you should update your + simTag accordingly. +2. objTag - uniquely identifies a particular object within a simulation. + This is the same across simulations, but should be a unique name for the + object within the simulation. +3. date-time string - all pickled files have an additional date-time string + appended to the file name to avoid any accidental overwriting of data. -.. automodule:: SimLogger.FigLogger - :members: - :undoc-members: - :show-inheritance: +With that, all data can be uniquely named and pickled to file. Pickling provides +an easy way to store and load files in python, and is the compression method +used in the SimLogger. -SimLogger.SimNotify module --------------------------- +This could be an example script -.. automodule:: SimLogger.SimNotify +.. code-block:: python + from SimLogger import SimLogger + simTag = 'sampleSimulation' + objTag = 'sampleArray' + sampleArray = [1,2,3,4] + SimLogger.saveObj(simTag, objTag, sampleArray, makeNote=True) + + # Outputs + # 2024-06-25 10:44:44,063 [INFO ] Logger Loaded + # 2024-06-25 10:44:44,064 [INFO ] OBJECT,sampleSimulation,sampleArray,data/obj/sampleSimulation_sampleArray_2024-06-25_10-44-44.pkl + # + # additionally a file is saved: data/obj/sampleSimulation_sampleArray_2024-06-25_10-44-44.pkl + +Which lays out the simTag for the simualtion, then defines an object we are going to save, +with the objTag and then stores it with the SimLogger. +This also generates an example.log file so that you can always refer somewhere to know exactly where the data has been stored. + +In future analysis, the data can be loaded as needed using the following +To reference the object you will need the unique id of the object, which is simTag_objTag + +.. code-block:: python + from SimLogger import SimLogger + sampleArrayLoaded = SimLogger.getObjectFromuniqueId('sampleSimulation_sampleArray') + print(sampleArrayLoaded) + + # Ouptuts + # [1, 2, 3, 4] + +Within this framework, various configurations can be changed: +- log file name +- folder where the pickled files get saved + +SimLogger.SimLogger Functions +----------------------------- + +.. automodule:: SimLogger.SimLogger :members: :undoc-members: :show-inheritance: + diff --git a/docs/source/SimNotify.rst b/docs/source/SimNotify.rst new file mode 100644 index 0000000..baed15d --- /dev/null +++ b/docs/source/SimNotify.rst @@ -0,0 +1,41 @@ +SimLogger.SimNotify module +========================== + +Often when simulations are running for an extended period of time on a cluster machine, +not to mention the time spent waiting for available resource, it is useful to get remote +updates to know how well a simulation is performing. This module seeks to solve that +problem while being integrated into the SimLogger system. Additionally, this uses +an openly and freely available system for remote web notifications notify-run + +In this package we don't reference the python interface for notify-run since configuration +can be finicky. Instead, we will directly send post requests to simplify the process. + +Installation +------------ + +Nofity runs on channels, that you can then easily subscribe to through the web interface. +To do this, go to https://notify.run/ and create a new channel. Copy the link that it +generates. This link will be the URL to go to and see any notifications. + +Using that link you can then send notifications: + +.. code-block:: python + + from SimLogger import SimNotify + endpoint = "paste-link-here" + SimNotify.sendNotification("Hello World", endpoint) + + +Which will post a Hello World message on the channel. + +Additionally, the notification, endpoint, and response code from the server are all +listed in the log file. + +SimLogger.FigLogger functions +----------------------------- + +.. automodule:: SimLogger.SimNotify + :members: + :undoc-members: + :show-inheritance: + diff --git a/docs/source/index.rst b/docs/source/index.rst index 81ff05d..a6d841c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -9,10 +9,10 @@ Welcome to SimLogger's documentation! .. toctree:: :maxdepth: 2 :caption: Contents: - + SimLogger - - + FigLogger + SimNotify A simplified method for logging and saving information in scientific computing. @@ -20,9 +20,12 @@ SimLogger seeks to simplify the process of saving and archiving data for scientific computing by handling all the file naming a storing for you. Archive your python objects simply and easiy, and never worry about overwriting files. -All data is defined with 3 key components +All data is defined with 3 key components + ``simTag`` - tag unique for each simulation (like naming the experiment) + ``objTag`` - tag unique for each object (like the variable within the experiments) + ``date-time`` - this is automatically added by SimLogger to help ensure unqiueness of files With that you can save your data easily! @@ -75,7 +78,7 @@ Development =========== install all dependencies (including linting and testing) with: -``pip install -e '.[lint,test]`` +``pip install -e '.[lint,test,doc]`` run tests with: