v2.03 - Nifty Prediction added to GUI

- Nifty-50 Gap up/down prediction added - cosmetic updates for GUI - version bumped - tab added for similar stock search - chromadb similar search in progress - pyinstaller build and requirements updated - fixed similar search in CLI mode - similar search added to GUI - Update requirements.txt - chromadb added to top - try catch added for importing chromadb to resolve pytest errors - Release page updated - README updated
pranjal-joshi · Oct 3, 2023 · 644cf6a · 644cf6a
1 parent 57759b5
commit 644cf6a
Show file tree

Hide file tree

Showing 12 changed files with 236 additions and 59 deletions.
diff --git a/.github/dependencies/sqlite3.dll b/.github/dependencies/sqlite3.dll
diff --git a/.github/workflows/workflow-build-matrix.yml b/.github/workflows/workflow-build-matrix.yml
@@ -24,7 +24,7 @@ jobs:
           - os: windows-latest
             TARGET: Windows
             CMD_BUILD: |
-              pyinstaller --onefile --icon=src\icon.ico src\screenipy.py  --hidden-import cmath --hidden-import talib.stream --hidden-import numpy --hidden-import pandas --hidden-import alive-progress --hidden-import alive_progress
+              pyinstaller --onefile --icon=src\icon.ico src\screenipy.py  --hidden-import cmath --hidden-import talib.stream --hidden-import numpy --hidden-import pandas --hidden-import alive-progress --hidden-import alive_progress --hidden-import chromadb
             DEP_BUILD: |
               python -m pip install --upgrade pip
               echo Installing TA-lib...
@@ -44,7 +44,7 @@ jobs:
           - os: ubuntu-20.04
             TARGET: Linux
             CMD_BUILD: |
-              pyinstaller --onefile --icon=src/icon.ico src/screenipy.py  --hidden-import cmath --hidden-import talib.stream --hidden-import numpy --hidden-import pandas --hidden-import alive-progress --hidden-import alive_progress
+              pyinstaller --onefile --icon=src/icon.ico src/screenipy.py  --hidden-import cmath --hidden-import talib.stream --hidden-import numpy --hidden-import pandas --hidden-import alive-progress --hidden-import alive_progress --hidden-import chromadb
               mv /home/runner/work/Screeni-py/Screeni-py/dist/screenipy /home/runner/work/Screeni-py/Screeni-py/dist/screenipy.bin
               chmod +x /home/runner/work/Screeni-py/Screeni-py/dist/screenipy.bin
             DEP_BUILD: |
@@ -68,7 +68,7 @@ jobs:
           - os: macos-latest
             TARGET: MacOS
             CMD_BUILD: |
-              pyinstaller --onefile --windowed --icon=src/icon.ico src/screenipy.py  --hidden-import cmath --hidden-import talib.stream --hidden-import numpy --hidden-import pandas --hidden-import alive-progress --hidden-import alive_progress
+              pyinstaller --onefile --windowed --icon=src/icon.ico src/screenipy.py  --hidden-import cmath --hidden-import talib.stream --hidden-import numpy --hidden-import pandas --hidden-import alive-progress --hidden-import alive_progress --hidden-import chromadb
               mv /Users/runner/work/Screeni-py/Screeni-py/dist/screenipy /Users/runner/work/Screeni-py/Screeni-py/dist/screenipy.run
             DEP_BUILD: |
               brew install ta-lib

diff --git a/README.md b/README.md
@@ -19,13 +19,24 @@
 
 ### A Python-based stock screener for NSE, India.
 
-**Screenipy** is an advanced stock screener to find potential breakout stocks from NSE and tell it's possible breakout values. It also helps to find the stocks which are consolidating and may breakout, or the particular chart patterns that you're looking specifically to make your decisions.
+**Screenipy** is an advanced stock screener to find potential breakout stocks from NSE and tell its possible breakout values. It also helps to find the stocks that are consolidating and may breakout, or the particular chart patterns that you're looking for specifically to make your decisions.
 Screenipy is totally customizable and it can screen stocks with the settings that you have provided.
 
-## How to use?
+## How to use? (New Version - GUI Based)
+* Install Docker Desktop and pull the `latest` docker image from the [release](https://github.com/pranjal-joshi/Screeni-py/releases/latest) page.
+* Checkout this [YouTube Video](https://youtu.be/2HMN0ac4H20) for detailed installation guide.
+
+<img width="1438" alt="image" src="https://github.com/pranjal-joshi/Screeni-py/assets/6128978/2016be00-5892-4735-8ab3-5f5b70add103">
+<img width="1438" alt="image" src="https://github.com/pranjal-joshi/Screeni-py/assets/6128978/28947290-7f42-4f6f-9fc0-0bae1ee6d6f4">
+<img width="1438" alt="image" src="https://github.com/pranjal-joshi/Screeni-py/assets/6128978/857f8acc-a4e8-4b86-a748-c26057b0e8b1">
+<img width="1438" alt="image" src="https://github.com/pranjal-joshi/Screeni-py/assets/6128978/360b5faa-f4f4-4df6-bec1-90985889bee6">
+<img width="1438" alt="image" src="https://github.com/pranjal-joshi/Screeni-py/assets/6128978/99903d67-d450-4c04-93ae-1f5bb4b905a5">
+
+
+## How to use? (Older Version - CLI Based)
 * Download the suitable file according to your OS or install Docker Desktop and pull the `latest` docker image.
-* Linux & Mac users should make sure that the `screenipy.bin or screenipy.run` is having `execute` permission.
-* **Run** the file. Following window will appear after a brief delay.
+* Linux & Mac users should make sure that the `screenipy.bin or screenipy.run` has `execute` permission.
+* **Run** the file. The following window will appear after a brief delay.
 
 ![home](https://raw.githubusercontent.com/pranjal-joshi/Screeni-py/new-features/screenshots/screenipy_demo.gif)
 
@@ -39,7 +50,7 @@ Screenipy is totally customizable and it can screen stocks with the settings tha
 ![results](https://raw.githubusercontent.com/pranjal-joshi/Screeni-py/new-features/screenshots/results.png)
 ![done](https://raw.githubusercontent.com/pranjal-joshi/Screeni-py/new-features/screenshots/done.png)
 
-* Once done, you can also save the results in an excel file.
+* Once done, you can also save the results in an Excel file.
 
 ## Understanding the Result Table:
 
@@ -49,12 +60,12 @@ The Result table contains a lot of different parameters which can be pretty over
 |:---:|:---:|:---|:---|
 |1|**Stock**|This is a NSE scrip symbol. If your OS/Terminal supports unicode, You can directly open **[TradingView](https://in.tradingview.com/)** charts by pressing `Ctrl+Click` on the stock name.|[TATAMOTORS](https://in.tradingview.com/chart?symbol=NSE%3ATATAMOTORS)|
 |2|**Consolidating**|It gives the price range in which stock is trading since last `N` days. `N` is configurable and can be modified by executing `Edit User Configuration` option.|If stock is trading between price 100-120 in last 30 days, Output will be `Range = 20.0 %`|
-|3|**Breakout (N Days)**|This is pure magic! The `BO` is Breakout level in last N days while `R` is the next resistance level if available. Investor should consider both BO & R level to decide entry/exits in their trades.|`B:302, R:313`(Breakout level is 100 & Next resistance is 102)|
+|3|**Breakout (N Days)**|This is pure magic! The `BO` is Breakout level in last N days while `R` is the next resistance level if available. An investor should consider both BO & R level to decide entry/exits in their trades.|`B:302, R:313`(Breakout level is 100 & Next resistance is 102)|
 |4|**LTP**|LTP is the Last Traded Price of an asset traded on NSE.|`298.7` (Stock is trading at this price)|
 |5|**Volume**|Volume shows the relative volume of the recent candle with respect to 20 period MA of Volume. It could be `Unknown` for newly listed stocks.|if 20MA(Volume) is 1M and todays Volume is 2.8M, then `Volume = 2.8x`|
-|6|**MA-Signal**|It describes the price trend of an asset by analysing various 50-200 MA/EMA crossover strategies.|`200MA-Support`,`BullCross-50MA` etc|
-|7|**RSI**|For the momentum traders, it describes 14-period RSI for quick decision making about their trading plans|`0 to 100`|
-|8|**Trend**|By using advance algorithms, the average trendlines are computed for `N` days and their strenght is displayed depending on steepness of trendlines. (This does NOT show any trendline on chart, it is calculated internally)|`Strong Up`, `Weak Down` etc.|
+|6|**MA-Signal**|It describes the price trend of an asset by analyzing various 50-200 MA/EMA crossover strategies.|`200MA-Support`,`BullCross-50MA` etc|
+|7|**RSI**|For the momentum traders, it describes 14-period RSI for quick decision-making about their trading plans|`0 to 100`|
+|8|**Trend**|By using advanced algorithms, the average trendlines are computed for `N` days and their strength is displayed depending on the steepness of the trendlines. (This does NOT show any trendline on a chart, it is calculated internally)|`Strong Up`, `Weak Down` etc.|
 |9|**Pattern**|If the chart or the candle itself forming any important pattern in the recent timeframe or as per the selected screening option, various important patterns will be indicated here.|`Momentum Gainer`, `Inside Bar (N)`,`Bullish Engulfing` etc.|
 
 ## Hack it your way:
@@ -73,7 +84,7 @@ cachestockdata = y
 onlystagetwostocks = y
 useema = n
 ```
-Try to tweak this parameters as per your trading styles. For example, If you're comfortable with weekly charts, make `duration=5d` and so on.
+Try to tweak these parameters as per your trading styles. For example, If you're comfortable with weekly charts, make `duration=5d` and so on.
 
 ## Installation Guide:
 
@@ -101,12 +112,12 @@ Try to tweak this parameters as per your trading styles. For example, If you're
 | Antivirus may block this as untrusted file ⚠️ | No issues with Antivirus | 
 | Need to download new file for every update | Updates quickly with minimal downloading |
 | No need of commands/technical knowledge | Very basic command execution skills may be required |
-| Custom configuration works and persisted accross the runs/restarts | Config file resets at every run but your custom config can be uploaded in GUI mode |
+| Custom configuration works and persists accross the runs/restarts | Config file resets at every run but your custom config can be uploaded in GUI mode |
 
-### How to setup and use Screeni-py with Docker?
+### How to set up and use Screeni-py with Docker?
 
 1. Download and Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) with default settings
-2. If you are using windows, update WSL (Windows subsystem for linux) by running `wsl --update` command in command prompt
+2. If you are using Windows, update WSL (Windows subsystem for linux) by running `wsl --update` command in the command prompt
 3. Restart your computer after installation
 4. Open Docker Desktop and keep it as it is
 5. Open Command Prompt (Windows) or Terminal (Mac/Linux) and run command `docker pull joshipranjal/screeni-py:latest`
@@ -119,8 +130,10 @@ Try to tweak this parameters as per your trading styles. For example, If you're
     docker run -it --entrypoint /bin/bash joshipranjal/screeni-py:latest -c "run_screenipy.sh --cli"
     ```
 
+Check out this [YouTube Video](https://youtu.be/2HMN0ac4H20) for a detailed installation guide.
+
 ## Contributing:
-* Please feel free to Suggest improvements bugs by creating an issue.
+* Please feel free to Suggest improvements/report bugs by creating an issue.
 * Please follow the [Guidelines for Contributing](https://github.com/pranjal-joshi/Screeni-py/blob/new-features/CONTRIBUTING.md) while making a Pull Request.
 
 ## Disclaimer:

diff --git a/requirements.txt b/requirements.txt
@@ -18,7 +18,7 @@ pyinstaller==5.6.2
 pytest-mock
 pytoml
 retrying
-scipy==1.7.3
+scipy==1.11.2
 ta-lib
 tabulate
 yfinance==0.1.87
@@ -61,4 +61,5 @@ pandas_ta
 # protobuf==3.19.6
 protobuf
 streamlit==1.26.0
-tensorflow
+tensorflow
+chromadb==0.4.10
diff --git a/src/classes/Changelog.py b/src/classes/Changelog.py
@@ -7,7 +7,7 @@
 
 from classes.ColorText import colorText
 
-VERSION = "2.02"
+VERSION = "2.03"
 
 changelog = colorText.BOLD + '[ChangeLog]\n' + colorText.END + colorText.BLUE + '''
 [1.00 - Beta]
@@ -216,4 +216,8 @@
 3. Cosmetic improvements
 4. YouTube Video added to docs
 
+[2.03]
+1. AI based Nifty-50 Gap up/down prediction added to GUI
+2. Cosmetic updates and minor bug-fixes.
+
 ''' + colorText.END
diff --git a/src/classes/OtaUpdater.py b/src/classes/OtaUpdater.py
@@ -85,7 +85,8 @@ def showWhatsNew():
         md = requests.get(url)
         txt = md.text
         txt = txt.split("New?")[1]
-        txt = txt.split("## Downloads")[0]
+        # txt = txt.split("## Downloads")[0]
+        txt = txt.split("## Installation Guide")[0]
         txt = txt.replace('**','').replace('`','').strip()
         return (txt+"\n")
 

diff --git a/src/classes/ParallelProcessing.py b/src/classes/ParallelProcessing.py
@@ -59,7 +59,7 @@ def run(self):
             sys.exit(0)
 
     def screenStocks(self, executeOption, reversalOption, maLength, daysForLowestVolume, minRSI, maxRSI, respChartPattern, insideBarToLookback, totalSymbols,
-                     configManager, fetcher, screener, candlePatterns, stock, newlyListedOnly, downloadOnly, printCounter=False):
+                     configManager, fetcher, screener, candlePatterns, stock, newlyListedOnly, downloadOnly, vectorSearch, printCounter=False):
         screenResults = pd.DataFrame(columns=[
             'Stock', 'Consolidating', 'Breaking-Out', 'MA-Signal', 'Volume', 'LTP', 'RSI', 'Trend', 'Pattern'])
         screeningDictionary = {'Stock': "", 'Consolidating': "",  'Breaking-Out': "",
@@ -105,6 +105,11 @@ def screenStocks(self, executeOption, reversalOption, maLength, daysForLowestVol
 
             fullData, processedData = screener.preprocessData(
                 data, daysToLookback=configManager.daysToLookback)
+
+            if type(vectorSearch) != bool and type(vectorSearch) and vectorSearch[2] == True:
+                executeOption = 0
+                with self.screenCounter.get_lock():
+                    screener.addVector(fullData, stock, vectorSearch[1])
 
             if newlyListedOnly:
                 if not screener.validateNewlyListed(fullData, period):

diff --git a/src/classes/Screener.py b/src/classes/Screener.py
@@ -9,16 +9,22 @@
 import math
 import numpy as np
 import pandas as pd
-# import talib
 import joblib
 import keras
+import time
 import classes.Utility as Utility
+from classes.Utility import isGui
 from sklearn.preprocessing import StandardScaler
 from scipy.signal import argrelextrema
 from scipy.stats import linregress
 from classes.ColorText import colorText
 from classes.SuppressOutput import SuppressOutput
 from classes.ScreenipyTA import ScreenerTA
+try:
+    import chromadb
+    CHROMA_AVAILABLE = True
+except:
+    CHROMA_AVAILABLE = False
 
 
 # Exception for newly listed stocks with candle nos < daysToLookback
@@ -592,9 +598,11 @@ def getNiftyPrediction(self, data, proxyServer):
             out = colorText.BOLD + colorText.GREEN + "BULLISH" + colorText.END + colorText.BOLD
             sug = "Stay Bullish!"
         if not Utility.tools.isClosingHour():
-            print(colorText.BOLD + colorText.WARN + "Note: The AI prediction should be executed After 3 PM or Near to Closing time as the Prediction Accuracy is based on the Closing price!" + colorText.END)
+            print(colorText.BOLD + colorText.WARN + "Note: The AI prediction should be executed After 3 PM Around the Closing hours as the Prediction Accuracy is based on the Closing price!" + colorText.END)
         print(colorText.BOLD + colorText.BLUE + "\n" + "[+] Nifty AI Prediction -> " + colorText.END + colorText.BOLD + "Market may Open {} next day! {}".format(out, sug) + colorText.END)
         print(colorText.BOLD + colorText.BLUE + "\n" + "[+] Nifty AI Prediction -> " + colorText.END + "Probability/Strength of Prediction = {}%".format(Utility.tools.getSigmoidConfidence(pred[0])))
+        if isGui():
+            return pred, 'BULLISH' if pred <= 0.5 else 'BEARISH', Utility.tools.getSigmoidConfidence(pred[0])
         return pred
 
     def monitorFiveEma(self, proxyServer, fetcher, result_df, last_signal, risk_reward = 3):
@@ -668,7 +676,27 @@ def monitorFiveEma(self, proxyServer, fetcher, result_df, last_signal, risk_rewa
         result_df.drop_duplicates(keep='last', inplace=True)
         result_df.sort_values(by='Time', inplace=True)
         return result_df[::-1]
-
+
+    # Add data to vector database
+    def addVector(self, data, stockCode, daysToLookback):
+        data = data[::-1] # Reinverting preprocessedData for pct_change
+        data = data.pct_change()
+        # data = data[::-1]     # Do we need to invert again? No we dont - See operation after flatten
+        data = data[['Open', 'High', 'Low', 'Close']]
+        data = data.reset_index(drop=True)
+        data = data.dropna()
+        data = data.to_numpy().flatten().tolist()
+        data = data[(-4 * daysToLookback):]     # Keep only OHLC * daysToLookback samples
+        if len(data) == (4 * daysToLookback):
+            chroma_client = chromadb.PersistentClient(path="./chromadb_store/")
+            collection = chroma_client.get_or_create_collection(name="nse_stocks")
+            collection.upsert(
+                embeddings=[data],
+                documents=[stockCode],
+                ids=[stockCode]
+            )
+            return data
+
 
     '''
     # Find out trend for days to lookback

diff --git a/src/classes/Utility.py b/src/classes/Utility.py
@@ -286,6 +286,19 @@ def promptChartPatterns():
             input(colorText.BOLD + colorText.FAIL +
                   "\n[+] Invalid Option Selected. Press Any Key to Continue..." + colorText.END)
             return (None, None)
+
+    # Prompt for Similar stock search
+    def promptSimilarStockSearch():
+        try:
+            stockCode = str(input(colorText.BOLD + colorText.WARN +
+                                    "\n[+] Enter the Name of the stock to search similar stocks for: " + colorText.END)).upper()
+            candles = int(input(colorText.BOLD + colorText.WARN +
+                                    "\n[+] How many candles (TimeFrame) to look back for similarity? : " + colorText.END))
+            return stockCode, candles
+        except ValueError:
+            input(colorText.BOLD + colorText.FAIL +
+                "\n[+] Invalid Option Selected. Press Any Key to Continue..." + colorText.END)
+            return None, None
 
     def getProgressbarStyle():
         bar = 'smooth'

diff --git a/src/release.md b/src/release.md
@@ -5,23 +5,23 @@ Celebrating more than 7K+ Downloads - Thank You for your support :tada:
 
 🐳 **Docker containers are now released for quick setup and easy usage!**
 
-1. New Index - **F&O Stocks Only** Added for F&O traders
-2. Trend detection fixed (#173) and % change added.
+1. **Search Similar Stocks** Added using Vector Similarity search - Try `Select Index for Screening > S` if still using CLI
+2. New Index - **F&O Stocks Only** Added for F&O traders with modified screening criterias.
 3. **Artificial Intelligence v2 for Nifty 50 Prediction** - Predict Next day Gap-up/down - Try `Select Index for Screening > N`
-4. **Live Intraday Scanner - 5 EMA** - Try `Select Index for Screening > E` - Get live trade entries for Nifty and BankNifty with notifications!
-5. New Screener **Buy at Trendline** added for Swing/Mid/Long term traders - Try `Option > 7 > 5`.
-6. Alternate Data Source for faster After-Market Analysis - Optimizations and  Cosmetic Updates!
+4. New Screener **Buy at Trendline** added for Swing/Mid/Long term traders - Try `Option > 7 > 5`.
+6. Alternate Data Source for faster After-Market Analysis - OTA Optimizations and Cosmetic Updates for Docker/GUI!
 
 ## Installation Guide
 
 [![Screeni-py - Detailed Installation Guide](https://markdown-videos-api.jorgenkh.no/url?url=https%3A%2F%2Fyoutu.be%2F2HMN0ac4H20)](https://youtu.be/2HMN0ac4H20)
 
-## Downloads
+## Downloads 
+#### Deprycated - Use Docker Method mentioned in next section
 | Operating System | Executable File |
 | :-: | --- |
-| ![Windows](https://img.shields.io/badge/Windows-0078D6?style=for-the-badge&logo=windows&logoColor=white) | **[screenipy.exe](https://github.com/pranjal-joshi/Screeni-py/releases/download/2.02/screenipy.exe)** |
-| ![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) | **[screenipy.bin](https://github.com/pranjal-joshi/Screeni-py/releases/download/2.02/screenipy.bin)** |
-| ![Mac OS](https://img.shields.io/badge/mac%20os-D3D3D3?style=for-the-badge&logo=apple&logoColor=000000) | **[screenipy.run](https://github.com/pranjal-joshi/Screeni-py/releases/download/2.02/screenipy.run)** ([Read Installation Guide](https://github.com/pranjal-joshi/Screeni-py/blob/main/INSTALLATION.md#for-macos)) |
+| ![Windows](https://img.shields.io/badge/Windows-0078D6?style=for-the-badge&logo=windows&logoColor=white) | **[screenipy.exe](https://github.com/pranjal-joshi/Screeni-py/releases/download/2.03/screenipy.exe)** |
+| ![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) | **[screenipy.bin](https://github.com/pranjal-joshi/Screeni-py/releases/download/2.03/screenipy.bin)** |
+| ![Mac OS](https://img.shields.io/badge/mac%20os-D3D3D3?style=for-the-badge&logo=apple&logoColor=000000) | **[screenipy.run](https://github.com/pranjal-joshi/Screeni-py/releases/download/2.03/screenipy.run)** ([Read Installation Guide](https://github.com/pranjal-joshi/Screeni-py/blob/main/INSTALLATION.md#for-macos)) |
 
 ## [Docker Releases](https://hub.docker.com/r/joshipranjal/screeni-py/tags)