Merge branch 'freqtrade-develop' of https://github.com/stash86/freqtrade

 into freqtrade-develop

.github/workflows/ci.yml

@@ -517,14 +517,14 @@ jobs:
           ft_client/dist
         retention-days: 10
 
-  deploy-pypi:
-    name: "Deploy to PyPI"
+  deploy-test-pypi:
+    name: "Publish Python 🐍 distribution 📦 to TestPyPI"
     needs: [ build ]
     runs-on: ubuntu-22.04
     if: (github.event_name == 'release')
     environment:
-      name: release
-      url: https://pypi.org/p/freqtrade
+      name: testpypi
+      url: https://test.pypi.org/p/freqtrade
     permissions:
       id-token: write
 
@@ -538,12 +538,33 @@ jobs:
         path: dist
         merge-multiple: true
 
-
     - name: Publish to PyPI (Test)
       uses: pypa/[email protected]
       with:
         repository-url: https://test.pypi.org/legacy/
 
+
+  deploy-pypi:
+    name: "Publish Python 🐍 distribution 📦 to PyPI"
+    needs: [ build ]
+    runs-on: ubuntu-22.04
+    if: (github.event_name == 'release')
+    environment:
+      name: pypi
+      url: https://pypi.org/p/freqtrade
+    permissions:
+      id-token: write
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Download artifact  📦
+      uses: actions/download-artifact@v4
+      with:
+        pattern: freqtrade*-build
+        path: dist
+        merge-multiple: true
+
     - name: Publish to PyPI
       uses: pypa/[email protected]
 

docs/configuration.md

@@ -225,6 +225,7 @@ Mandatory parameters are marked as **Required**, which means that they are requi
 | `exchange.skip_open_order_update` | Skips open order updates on startup should the exchange cause problems. Only relevant in live conditions.<br>*Defaults to `false`*<br> **Datatype:** Boolean
 | `exchange.unknown_fee_rate` | Fallback value to use when calculating trading fees. This can be useful for exchanges which have fees in non-tradable currencies. The value provided here will be multiplied with the "fee cost".<br>*Defaults to `None`<br> **Datatype:** float
 | `exchange.log_responses` | Log relevant exchange responses. For debug mode only - use with care.<br>*Defaults to `false`*<br> **Datatype:** Boolean
+| `exchange.only_from_ccxt` | Prevent data-download from data.binance.vision. Leaving this as false can greatly speed up downloads, but may be problematic if the site is not available.<br>*Defaults to `false`*<br> **Datatype:** Boolean
 | `experimental.block_bad_exchanges` | Block exchanges known to not work with freqtrade. Leave on default unless you want to test if that exchange works now. <br>*Defaults to `true`.* <br> **Datatype:** Boolean
 | | **Plugins**
 | `edge.*` | Please refer to [edge configuration document](edge.md) for detailed explanation of all possible configuration options.

docs/data-download.md

@@ -89,7 +89,7 @@ Common arguments:
 
 !!! Tip "Downloading all data for one quote currency"
     Often, you'll want to download data for all pairs of a specific quote-currency. In such cases, you can use the following shorthand:
-    `freqtrade download-data --exchange binance --pairs .*/USDT <...>`. The provided "pairs" string will be expanded to contain all active pairs on the exchange.
+    `freqtrade download-data --exchange binance --pairs ".*/USDT" <...>`. The provided "pairs" string will be expanded to contain all active pairs on the exchange.
     To also download data for inactive (delisted) pairs, add `--include-inactive-pairs` to the command.
 
 !!! Note "Startup period"
@@ -116,7 +116,7 @@ freqtrade download-data --exchange binance --pairs ETH/USDT XRP/USDT BTC/USDT
 or as regex (in this case, to download all active USDT pairs)
 
 ```bash
-freqtrade download-data --exchange binance --pairs .*/USDT
+freqtrade download-data --exchange binance --pairs ".*/USDT"
 ```
 
 ### Other Notes

docs/exchanges.md

@@ -269,6 +269,7 @@ On startup, freqtrade will set the position mode to "One-way Mode" for the whole
 As bybit doesn't provide funding rate history, the dry-run calculation is used for live trades as well.
 
 API Keys for live futures trading must have the following permissions:
+
 * Read-write
 * Contract - Orders
 * Contract - Positions
@@ -308,20 +309,20 @@ It's therefore required to pass the UID as well.
 !!! Tip "Stoploss on Exchange"
     Hyperliquid supports `stoploss_on_exchange` and uses `stop-loss-limit` orders. It provides great advantages, so we recommend to benefit from it.
 
-Hyperliquid is a Decentralized Exchange (DEX). Decentralized exchanges work a bit different compared to normal exchanges. Instead of authenticating private API calls using an API key, private API calls need to be signed with the private key of your wallet.
+Hyperliquid is a Decentralized Exchange (DEX). Decentralized exchanges work a bit different compared to normal exchanges. Instead of authenticating private API calls using an API key, private API calls need to be signed with the private key of your wallet (We recommend using an api Wallet for this, generated either on Hyperliquid or in your wallet of choice).
 This needs to be configured like this:
 
 ```json
 "exchange": {
     "name": "hyperliquid",
     "walletAddress": "your_eth_wallet_address",
-    "privateKey": "your_private_key",
+    "privateKey": "your_api_private_key",
     // ...
 }
 ```
 
-* walletAddress must be in hex format: `0x<40 hex characters>`, and can be easily copied from your wallet.
-* privateKey also must be in hex format: `0x<64 hex characters>`, and can either be exported from your wallet or regenerated using your mnemonic phrase.
+* walletAddress in hex format: `0x<40 hex characters>` - Can be easily copied from your wallet - and should be your wallet address, not your API Wallet Address.
+* privateKey in hex format: `0x<64 hex characters>` - Use the key the API Wallet shows on creation.
 
 Hyperliquid handles deposits and withdrawals on the Arbitrum One chain, a Layer 2 scaling solution built on top of Ethereum. Hyperliquid uses USDC as quote / collateral. The process of depositing USDC on Hyperliquid requires a couple of steps, see [how to start trading](https://hyperliquid.gitbook.io/hyperliquid-docs/onboarding/how-to-start-trading) for details on what steps are needed.
 
@@ -330,15 +331,14 @@ Hyperliquid handles deposits and withdrawals on the Arbitrum One chain, a Layer
     Unfortunately, hyperliquid only offers 5000 historic candles, so backtesting will either need to build candles historically (by waiting and downloading the data incrementally over time) - or will be limited to the last 5000 candles.
 
 !!! Info "Some general best practices (non exhaustive)"
-    * Beware of supply chain attacks, like pip package poisoning etcetera. However you export or (re-)generate your private key, make sure your environment is safe.
-    * Interact as little with the private key as possible. Store it in a separate file from the config.json (secrets.json for example) that you never have to touch, and secure it.
+    * Beware of supply chain attacks, like pip package poisoning etcetera. Whenever you use your private key, make sure your environment is safe.
+    * Don't use your actual wallet private key for trading. Use the Hyperliquid [API generator](https://app.hyperliquid.xyz/API) to create a separate API wallet.
+    * Don't store your actual wallet private key on the server you use for freqtrade. Use the API wallet private key instead. This key won't allow withdrawals, only trading.
     * Always keep your mnemonic phrase and private key private.
     * Don't use the same mnemonic as the one you had to backup when initializing a hardware wallet, using the same mnemonic basically deletes the security of your hardware wallet.
-    * Create a different software wallet, only transfer the funds you want to trade with to that wallet, and use that wallet / private key to trade on Hyperliquid.
-    * Remember that if someone hacks the host you use for trading, or any other host you stored your private key / mnemonic on, you will lose the funds protected by that private key. That means the funds on that wallet and the funds deposited on Hyperliquid.
+    * Create a different software wallet, only transfer the funds you want to trade with to that wallet, and use that wallet to trade on Hyperliquid.
     * If you have funds you don't want to use for trading (after making a profit for example), transfer them back to your hardware wallet.
 
-
 ## All exchanges
 
 Should you experience constant errors with Nonce (like `InvalidNonce`), it is best to regenerate the API keys. Resetting Nonce is difficult and it's usually easier to regenerate the API keys.
@@ -348,7 +348,7 @@ Should you experience constant errors with Nonce (like `InvalidNonce`), it is be
 * The Ocean (exchange id: `theocean`) exchange uses Web3 functionality and requires `web3` python package to be installed:
 
 ```shell
-$ pip3 install web3
+pip3 install web3
 ```
 
 ### Getting latest price / Incomplete candles

docs/requirements-docs.txt

@@ -1,6 +1,6 @@
 markdown==3.7
 mkdocs==1.6.1
-mkdocs-material==9.5.45
+mkdocs-material==9.5.47
 mdx_truly_sane_lists==1.3
 pymdown-extensions==10.12
 jinja2==3.1.4

freqtrade/__init__.py

@@ -1,6 +1,6 @@
 """Freqtrade bot"""
 
-__version__ = "2024.11-dev"
+__version__ = "2024.12-dev"
 
 if "dev" in __version__:
     from pathlib import Path

freqtrade/data/history/history_utils.py

@@ -285,6 +285,7 @@ def _download_pair_history(
             candle_type=candle_type,
             until_ms=until_ms if until_ms else None,
         )
+        logger.info(f"Downloaded data for {pair} with length {len(new_dataframe)}.")
         if data.empty:
             data = new_dataframe
         else:
@@ -603,7 +604,7 @@ def download_data(
     Download data function. Used from both cli and API.
     """
     timerange = TimeRange()
-    if "days" in config:
+    if "days" in config and config["days"] is not None:
         time_since = (datetime.now() - timedelta(days=config["days"])).strftime("%Y%m%d")
         timerange = TimeRange.parse_timerange(f"{time_since}-")
 

freqtrade/exchange/binance.py

@@ -5,13 +5,18 @@
 from pathlib import Path
 
 import ccxt
+from pandas import DataFrame
 
+from freqtrade.constants import DEFAULT_DATAFRAME_COLUMNS
 from freqtrade.enums import CandleType, MarginMode, PriceType, TradingMode
 from freqtrade.exceptions import DDosProtection, OperationalException, TemporaryError
 from freqtrade.exchange import Exchange
+from freqtrade.exchange.binance_public_data import concat_safe, download_archive_ohlcv
 from freqtrade.exchange.common import retrier
-from freqtrade.exchange.exchange_types import FtHas, OHLCVResponse, Tickers
+from freqtrade.exchange.exchange_types import FtHas, Tickers
+from freqtrade.exchange.exchange_utils_timeframe import timeframe_to_msecs
 from freqtrade.misc import deep_merge_dicts, json_load
+from freqtrade.util.datetime_helpers import dt_from_ts, dt_ts
 
 
 logger = logging.getLogger(__name__)
@@ -102,41 +107,114 @@ def additional_exchange_init(self) -> None:
         except ccxt.BaseError as e:
             raise OperationalException(e) from e
 
-    async def _async_get_historic_ohlcv(
+    def get_historic_ohlcv(
         self,
         pair: str,
         timeframe: str,
         since_ms: int,
         candle_type: CandleType,
         is_new_pair: bool = False,
-        raise_: bool = False,
         until_ms: int | None = None,
-    ) -> OHLCVResponse:
+    ) -> DataFrame:
         """
         Overwrite to introduce "fast new pair" functionality by detecting the pair's listing date
         Does not work for other exchanges, which don't return the earliest data when called with "0"
         :param candle_type: Any of the enum CandleType (must match trading mode!)
         """
         if is_new_pair:
-            x = await self._async_get_candle_history(pair, timeframe, candle_type, 0)
+            x = self.loop.run_until_complete(
+                self._async_get_candle_history(pair, timeframe, candle_type, 0)
+            )
             if x and x[3] and x[3][0] and x[3][0][0] > since_ms:
                 # Set starting date to first available candle.
                 since_ms = x[3][0][0]
                 logger.info(
                     f"Candle-data for {pair} available starting with "
                     f"{datetime.fromtimestamp(since_ms // 1000, tz=timezone.utc).isoformat()}."
                 )
+                if until_ms and since_ms >= until_ms:
+                    logger.warning(
+                        f"No available candle-data for {pair} before "
+                        f"{dt_from_ts(until_ms).isoformat()}"
+                    )
+                    return DataFrame(columns=DEFAULT_DATAFRAME_COLUMNS)
+
+        if (
+            self._config["exchange"].get("only_from_ccxt", False)
+            or
+            # only download timeframes with significant improvements,
+            # otherwise fall back to rest API
+            not (
+                (candle_type == CandleType.SPOT and timeframe in ["1s", "1m", "3m", "5m"])
+                or (
+                    candle_type == CandleType.FUTURES
+                    and timeframe in ["1m", "3m", "5m", "15m", "30m"]
+                )
+            )
+        ):
+            return super().get_historic_ohlcv(
+                pair=pair,
+                timeframe=timeframe,
+                since_ms=since_ms,
+                candle_type=candle_type,
+                is_new_pair=is_new_pair,
+                until_ms=until_ms,
+            )
+        else:
+            # Download from data.binance.vision
+            return self.get_historic_ohlcv_fast(
+                pair=pair,
+                timeframe=timeframe,
+                since_ms=since_ms,
+                candle_type=candle_type,
+                is_new_pair=is_new_pair,
+                until_ms=until_ms,
+            )
 
-        return await super()._async_get_historic_ohlcv(
-            pair=pair,
-            timeframe=timeframe,
-            since_ms=since_ms,
-            is_new_pair=is_new_pair,
-            raise_=raise_,
-            candle_type=candle_type,
-            until_ms=until_ms,
+    def get_historic_ohlcv_fast(
+        self,
+        pair: str,
+        timeframe: str,
+        since_ms: int,
+        candle_type: CandleType,
+        is_new_pair: bool = False,
+        until_ms: int | None = None,
+    ) -> DataFrame:
+        """
+        Fastly fetch OHLCV data by leveraging https://data.binance.vision.
+        """
+        df = self.loop.run_until_complete(
+            download_archive_ohlcv(
+                candle_type=candle_type,
+                pair=pair,
+                timeframe=timeframe,
+                since_ms=since_ms,
+                until_ms=until_ms,
+                markets=self.markets,
+            )
         )
 
+        # download the remaining data from rest API
+        if df.empty:
+            rest_since_ms = since_ms
+        else:
+            rest_since_ms = dt_ts(df.iloc[-1].date) + timeframe_to_msecs(timeframe)
+
+        # make sure since <= until
+        if until_ms and rest_since_ms > until_ms:
+            rest_df = DataFrame()
+        else:
+            rest_df = super().get_historic_ohlcv(
+                pair=pair,
+                timeframe=timeframe,
+                since_ms=rest_since_ms,
+                candle_type=candle_type,
+                is_new_pair=is_new_pair,
+                until_ms=until_ms,
+            )
+        all_df = concat_safe([df, rest_df])
+        return all_df
+
     def funding_fee_cutoff(self, open_date: datetime):
         """
         Funding fees are only charged at full hours (usually every 4-8h).