-
Notifications
You must be signed in to change notification settings - Fork 3
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
7a4846d
commit fc0092f
Showing
3 changed files
with
331 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
# Getting started with thread | ||
|
||
Thanks for using thread! I hope you find it useful for your projects. | ||
|
||
Here's to you get started. | ||
|
||
--- | ||
|
||
## Prerequisites | ||
|
||
* Python 3.11+ | ||
|
||
The project is quite heavily type-annotated, and we use `Concatenate[Any, ...]` in some function declarations. | ||
However `Python <=3.10` does not support `...` being the last argument as laid out in [this stack overflow question](https://stackoverflow.com/questions/74893354/is-literal-ellipsis-really-valid-as-paramspec-last-argument). | ||
|
||
If possible, I may release a sister version of thread that is compatible with `Python 3.9+` in the future, but for the time being, | ||
support will extend only from Python 3.11+ | ||
|
||
<br /> | ||
|
||
|
||
## Installing | ||
|
||
### From pip (Recommended) | ||
```sh | ||
pip install thread | ||
``` | ||
|
||
### Building from source (Not Recommended) | ||
```sh | ||
# Clone this repository | ||
git clone https://github.com/caffeine-addictt/thread | ||
|
||
# Install dependencies | ||
pip install poetry | ||
|
||
# Build the distribution | ||
python3 -m poetry build | ||
|
||
# Install the distribution | ||
pip install -e . | ||
``` | ||
|
||
<br /> | ||
|
||
|
||
## Importing thread | ||
|
||
Import thread into your .py file | ||
```py | ||
import thread | ||
``` | ||
|
||
Now you have successfully installed thread! | ||
|
||
[See here](./threading.md) for how to using the `thread.Thread` class! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,140 @@ | ||
# Parallel Processing Documentation | ||
|
||
I will lay out how to use the `thread.ParallelProcessing` class! | ||
|
||
<br /> | ||
<details> | ||
<summary>Jump to</summary> | ||
<ul> | ||
<li><a href='#importing-the-class'> Import the class</a></li> | ||
<li><a href='#initializing-a-thread'> Initialize a thread </a></li> | ||
</ul> | ||
</details> | ||
|
||
|
||
Don't have the thread library? [See here](./getting-started.md) for installing thread | ||
|
||
--- | ||
|
||
## Importing the class | ||
|
||
```py | ||
from thread import ParallelProcessing | ||
``` | ||
|
||
<br /> | ||
|
||
|
||
## How does it work? | ||
|
||
Parallel Processing works best by optimizing data processing with large datasets. | ||
|
||
What it does: | ||
```py | ||
dataset = [1, 2, 3, ..., 2e10] | ||
|
||
# Splits into chunks as evenly as possible | ||
# thread_count = min(max_threads, len(dataset)) | ||
# n == len(chunks) == len(thread_count) | ||
chunks = [[1, 2, 3, ...], [50, 51, 52, ...], ...] | ||
|
||
# Initialize and run n threads | ||
# each thread handles 1 chunk of data and parses it into the function | ||
|
||
# processed data is arranged back in order | ||
|
||
# processed data is returned as a list[Data_Out] | ||
``` | ||
|
||
<br /> | ||
|
||
|
||
## Initializing a parallel process | ||
|
||
A simple example | ||
```py | ||
def my_data_processor(Data_In) -> Data_Out: ... | ||
|
||
# Reccommended way | ||
my_processor = ParallelProcessing( | ||
function = my_data_processor, | ||
dataset = [i in range(0, n)] | ||
) | ||
|
||
# OR | ||
# Not the reccommended way | ||
my_processor = ParallelProcessing(my_data_processor, [i in range(0, n)]) | ||
``` | ||
|
||
It can be ran by invoking the `start()` method | ||
```py | ||
my_processor.start() | ||
``` | ||
|
||
> [!NOTE] | ||
> The **threading.ParallelProcessing()** class from python will only be initialized when **start()** is invoked | ||
<br /> | ||
|
||
|
||
### Parameters | ||
|
||
* function : (DataProcessor, dataset, *args, **kwargs) -> Any | Data_Out | ||
> This should be a function that takes in a dataset and/or anything and returns Data_Out and/or anything | ||
* dataset : Sequence[Data_In] = () | ||
> This should be an interable sequence of arguments parsed to the `DataProcessor` function<br /> | ||
> (e.g. tuple('foo', 'bar')) | ||
* *overflow_args : Overflow_In | ||
> These are arguments parsed to [**thread.Thread**](./threading.md#parameters) | ||
* **overflow_kwargs : Overflow_In | ||
> These are arguments parsed to [**thread.Thread**](./threading.md#parameters)<br /> | ||
> [!NOTE] | ||
> If `args` is present, then it will automatically be removed from kwargs and joined with `overflow_args` | ||
* **Raises** AssertionError: max_threads is invalid | ||
|
||
<br /> | ||
|
||
|
||
### Attributes | ||
|
||
These are attributes of [`ParallelProcessing`](#importing-the-class) class | ||
|
||
* results : List[Data_Out] | ||
> The result value | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
> **Raises** [`ThreadNotRunningError`](./exceptions.md#threadnotrunningerror) | ||
> **Raises** [`ThreadStillRunningError`](./exceptions.md#threadStillRunningError) | ||
<br /> | ||
|
||
|
||
### Methods | ||
|
||
These are methods of [`ParallelProcessing`](#importing-the-class) class | ||
|
||
* start : () -> None | ||
> Initializes the threads and starts it<br /> | ||
> **Raises** [`ThreadStillRunningError`](./exceptions.md#threadStillRunningError) | ||
* is_alive : () -> bool | ||
> Indicates whether the thread is still alive<br /> | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
* get_return_values : () -> Data_Out | ||
> Halts the current thread execution until the thread completes | ||
* join : () -> JoinTerminatedStatus | ||
> Halts the current thread execution until a thread completes or exceeds the timeout | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
> **Raises** [`ThreadNotRunningError`](./exceptions.md#threadnotrunningerror) | ||
<br /> | ||
|
||
|
||
Now you know how to use the [`Thread`](#importing-the-class) class! | ||
|
||
[See here](./parallel-processing.md) for how to using the `thread.ParallelProcessing` class! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
# Thread Class documentation | ||
|
||
I will lay out how to use the `thread.Thread` class! | ||
|
||
<br /> | ||
<details> | ||
<summary>Jump to</summary> | ||
<ul> | ||
<li><a href='#importing-the-class'> Import the class</a></li> | ||
<li><a href='#initializing-a-thread'> Initialize a thread </a></li> | ||
</ul> | ||
</details> | ||
|
||
|
||
Don't have the thread library? [See here](./getting-started.md) for installing thread | ||
|
||
--- | ||
|
||
## Importing the class | ||
|
||
```py | ||
from thread import Thread | ||
``` | ||
|
||
<br /> | ||
|
||
|
||
## Initializing a thread | ||
|
||
A simple thread can be prepared to be initialized with this | ||
```py | ||
def my_target(): ... | ||
|
||
# Reccommended way | ||
my_thread = Thread( | ||
target = my_target | ||
) | ||
|
||
# OR | ||
# Not the reccommended way | ||
my_thread = Thread(my_target) | ||
``` | ||
|
||
A thread can be ran by invoking the `start()` method | ||
```py | ||
my_thread.start() | ||
``` | ||
|
||
> [!NOTE] | ||
> The **threading.Thread()** class from python will only be initialized when **start()** is invoked | ||
<br /> | ||
|
||
|
||
### Parameters | ||
|
||
* target : (Data_In, *args, **kwargs) -> Any | Data_Out | ||
> This should be a function that takes in anything and returns anything | ||
* args : Sequence[Data_In] = () | ||
> This should be an interable sequence of arguments parsed to the `target` function <br /> | ||
> (e.g. tuple('foo', 'bar')) | ||
* kwargs : Mapping[str, Data_In] = {} | ||
> This should be the kwargs pased to the `target` function<br /> | ||
> (e.g. dict(foo = 'bar')) | ||
* ignore_errors : Sequence[type[Exception]] = () | ||
> This should be an interable sequence of all exceptions to ignore.<br /> | ||
> To ignore all exceptions, parse tuple(Exception) | ||
* suppress_errors : bool = False | ||
> This should be a boolean indicating whether exceptions will be raised.<br /> | ||
> If true, exceptions will only write to internal `errors` property<br /> | ||
> If false, exceptions will propagate if not ignored | ||
* name : Optional[str] = None | ||
> This is an argument parsed to `threading.Thread` | ||
* daemon : bool = False | ||
> This is an argument parsed to `threading.Thread` | ||
* *overflow_args : Overflow_In | ||
> These are arguments parsed to `threading.Thread` | ||
* **overflow_kwargs : Overflow_In | ||
> These are arguments parsed to `threading.Thread` | ||
<br /> | ||
|
||
|
||
### Attributes | ||
|
||
These are attributes of [`Thread`](#importing-the-class) class | ||
|
||
* result : Data_Out | ||
> The result value of the thread | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
> **Raises** [`ThreadNotRunningError`](./exceptions.md#threadnotrunningerror) | ||
> **Raises** [`ThreadStillRunningError`](./exceptions.md#threadStillRunningError) | ||
<br /> | ||
|
||
|
||
### Methods | ||
|
||
These are methods of [`Thread`](#importing-the-class) class | ||
|
||
* start : () -> None | ||
> Initializes the thread and starts it<br /> | ||
> **Raises** [`ThreadStillRunningError`](./exceptions.md#threadStillRunningError) | ||
* is_alive : () -> bool | ||
> Indicates whether the thread is still alive<br /> | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
* add_hook : ((Data_Out) -> Any | None) -> None | ||
> Hooks will be automatically invoked after a thread successfully completes, parsing the return value as the first argument | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
> **Raises** [`ThreadNotRunningError`](./exceptions.md#threadnotrunningerror) | ||
* get_return_value : () -> Data_Out | ||
> Halts the current thread execution until the thread completes | ||
* join : () -> JoinTerminatedStatus | ||
> Halts the current thread execution until a thread completes or exceeds the timeout | ||
> **Raises** [`ThreadNotInitializedError`](./exceptions.md#threadNotInitializedError) | ||
> **Raises** [`ThreadNotRunningError`](./exceptions.md#threadnotrunningerror) | ||
<br /> | ||
|
||
|
||
Now you know how to use the [`Thread`](#importing-the-class) class! | ||
|
||
[See here](./parallel-processing.md) for how to using the `thread.ParallelProcessing` class! |