Skip to content

Commit

Permalink
Merge branch 'freqtrade:develop' into freqtrade-develop
Browse files Browse the repository at this point in the history
  • Loading branch information
stash86 authored Jan 31, 2024
2 parents 951754b + 0bc5c87 commit 22f2f9d
Show file tree
Hide file tree
Showing 19 changed files with 492 additions and 61 deletions.
18 changes: 10 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -505,9 +505,10 @@ jobs:
python-version: "3.11"

- name: Extract branch name
shell: bash
run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF##*/})"
id: extract_branch
id: extract-branch
run: |
echo "GITHUB_REF='${GITHUB_REF}'"
echo "branch=${GITHUB_REF##*/}" >> "$GITHUB_OUTPUT"
- name: Dockerhub login
env:
Expand Down Expand Up @@ -536,7 +537,7 @@ jobs:

- name: Build and test and push docker images
env:
BRANCH_NAME: ${{ steps.extract_branch.outputs.branch }}
BRANCH_NAME: ${{ steps.extract-branch.outputs.branch }}
run: |
build_helpers/publish_docker_multi.sh
Expand All @@ -553,9 +554,10 @@ jobs:
- uses: actions/checkout@v4

- name: Extract branch name
shell: bash
run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF##*/})"
id: extract_branch
id: extract-branch
run: |
echo "GITHUB_REF='${GITHUB_REF}'"
echo "branch=${GITHUB_REF##*/}" >> "$GITHUB_OUTPUT"
- name: Dockerhub login
env:
Expand All @@ -566,7 +568,7 @@ jobs:
- name: Build and test and push docker images
env:
BRANCH_NAME: ${{ steps.extract_branch.outputs.branch }}
BRANCH_NAME: ${{ steps.extract-branch.outputs.branch }}
GHCR_USERNAME: ${{ github.actor }}
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker_update_readme.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Docker Hub Description
uses: peter-evans/dockerhub-description@v3
uses: peter-evans/dockerhub-description@v4
env:
DOCKERHUB_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKERHUB_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/pre-commit-update.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
add-paths: .pre-commit-config.yaml
labels: |
Tech maintenance
Dependencies
branch: update/pre-commit-hooks
title: Update pre-commit hooks
commit-message: "chore: update pre-commit hooks"
Expand Down
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ repos:
additional_dependencies:
- types-cachetools==5.3.0.7
- types-filelock==3.2.7
- types-requests==2.31.0.20240106
- types-requests==2.31.0.20240125
- types-tabulate==0.9.0.20240106
- types-python-dateutil==2.8.19.20240106
- SQLAlchemy==2.0.25
Expand All @@ -31,7 +31,7 @@ repos:

- repo: https://github.com/charliermarsh/ruff-pre-commit
# Ruff version.
rev: 'v0.1.14'
rev: 'v0.1.15'
hooks:
- id: ruff

Expand Down
61 changes: 59 additions & 2 deletions docs/includes/pairlists.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ In your configuration, you can use Static Pairlist (defined by the [`StaticPairL

Additionally, [`AgeFilter`](#agefilter), [`PrecisionFilter`](#precisionfilter), [`PriceFilter`](#pricefilter), [`ShuffleFilter`](#shufflefilter), [`SpreadFilter`](#spreadfilter) and [`VolatilityFilter`](#volatilityfilter) act as Pairlist Filters, removing certain pairs and/or moving their positions in the pairlist.

If multiple Pairlist Handlers are used, they are chained and a combination of all Pairlist Handlers forms the resulting pairlist the bot uses for trading and backtesting. Pairlist Handlers are executed in the sequence they are configured. You should always configure either `StaticPairList` or `VolumePairList` as the starting Pairlist Handler.
If multiple Pairlist Handlers are used, they are chained and a combination of all Pairlist Handlers forms the resulting pairlist the bot uses for trading and backtesting. Pairlist Handlers are executed in the sequence they are configured. You can define either `StaticPairList`, `VolumePairList`, `ProducerPairList`, `RemotePairList` or `MarketCapPairList` as the starting Pairlist Handler.

Inactive markets are always removed from the resulting pairlist. Explicitly blacklisted pairs (those in the `pair_blacklist` configuration setting) are also always removed from the resulting pairlist.

Expand All @@ -24,6 +24,7 @@ You may also use something like `.*DOWN/BTC` or `.*UP/BTC` to exclude leveraged
* [`VolumePairList`](#volume-pair-list)
* [`ProducerPairList`](#producerpairlist)
* [`RemotePairList`](#remotepairlist)
* [`MarketCapPairList`](#marketcappairlist)
* [`AgeFilter`](#agefilter)
* [`FullTradesFilter`](#fulltradesfilter)
* [`OffsetFilter`](#offsetfilter)
Expand Down Expand Up @@ -192,7 +193,8 @@ The RemotePairList is defined in the pairlists section of the configuration sett
"refresh_period": 1800,
"keep_pairlist_on_failure": true,
"read_timeout": 60,
"bearer_token": "my-bearer-token"
"bearer_token": "my-bearer-token",
"save_to_file": "user_data/filename.json"
}
]
```
Expand All @@ -207,6 +209,42 @@ In "append" mode, the retrieved pairlist is added to the original pairlist. All

The `pairlist_url` option specifies the URL of the remote server where the pairlist is located, or the path to a local file (if file:/// is prepended). This allows the user to use either a remote server or a local file as the source for the pairlist.

The `save_to_file` option, when provided with a valid filename, saves the processed pairlist to that file in JSON format. This option is optional, and by default, the pairlist is not saved to a file.

??? Example "Multi bot with shared pairlist example"

`save_to_file` can be used to save the pairlist to a file with Bot1:

```json
"pairlists": [
{
"method": "RemotePairList",
"mode": "whitelist",
"pairlist_url": "https://example.com/pairlist",
"number_assets": 10,
"refresh_period": 1800,
"keep_pairlist_on_failure": true,
"read_timeout": 60,
"save_to_file": "user_data/filename.json"
}
]
```

This saved pairlist file can be loaded by Bot2, or any additional bot with this configuration:

```json
"pairlists": [
{
"method": "RemotePairList",
"mode": "whitelist",
"pairlist_url": "file:///user_data/filename.json",
"number_assets": 10,
"refresh_period": 10,
"keep_pairlist_on_failure": true,
}
]
```

The user is responsible for providing a server or local file that returns a JSON object with the following structure:

```json
Expand All @@ -227,6 +265,25 @@ The optional `bearer_token` will be included in the requests Authorization Heade
!!! Note
In case of a server error the last received pairlist will be kept if `keep_pairlist_on_failure` is set to true, when set to false a empty pairlist is returned.

#### MarketCapPairList

`MarketCapPairList` employs sorting/filtering of pairs by their marketcap rank based of CoinGecko. It will only recognize coins up to the coin placed at rank 250. The returned pairlist will be sorted based of their marketcap ranks.

```json
"pairlists": [
{
"method": "MarketCapPairList",
"number_assets": 20,
"max_rank": 50,
"refresh_period": 86400
}
]
```

`number_assets` defines the maximum number of pairs returned by the pairlist. `max_rank` will determine the maximum rank used in creating/filtering the pairlist. It's expected that some coins within the top `max_rank` marketcap will not be included in the resulting pairlist since not all pairs will have active trading pairs in your preferred market/stake/exchange combination.

`refresh_period` setting defines the period (in seconds) at which the marketcap rank data will be refreshed. Defaults to 86,400s (1 day). The pairlist cache (`refresh_period`) is applicable on both generating pairlists (first position in the list) and filtering instances (not the first position in the list).

#### AgeFilter

Removes pairs that have been listed on the exchange for less than `min_days_listed` days (defaults to `10`) or more than `max_days_listed` days (defaults `None` mean infinity).
Expand Down
2 changes: 1 addition & 1 deletion docs/requirements-docs.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
markdown==3.5.2
mkdocs==1.5.3
mkdocs-material==9.5.4
mkdocs-material==9.5.6
mdx_truly_sane_lists==1.3
pymdown-extensions==10.7
jinja2==3.1.3
4 changes: 2 additions & 2 deletions docs/strategy-customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -1009,8 +1009,8 @@ This is a common pain-point, which can cause huge differences between backtestin

The following lists some common patterns which should be avoided to prevent frustration:

- don't use `shift(-1)`. This uses data from the future, which is not available.
- don't use `.iloc[-1]` or any other absolute position in the dataframe, this will be different between dry-run and backtesting.
- don't use `shift(-1)` or other negative values. This uses data from the future in backtesting, which is not available in dry or live modes.
- don't use `.iloc[-1]` or any other absolute position in the dataframe within `populate_` functions, as this will be different between dry-run and backtesting. Absolute `iloc` indexing is safe to use in callbacks however - see [Strategy Callbacks](strategy-callbacks.md).
- don't use `dataframe['volume'].mean()`. This uses the full DataFrame for backtesting, including data from the future. Use `dataframe['volume'].rolling(<window>).mean()` instead
- don't use `.resample('1h')`. This uses the left border of the interval, so moves data from an hour to the start of the hour. Use `.resample('1h', label='right')` instead.

Expand Down
2 changes: 1 addition & 1 deletion freqtrade/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
""" Freqtrade bot """
__version__ = '2024.1-dev'
__version__ = '2024.2-dev'

if 'dev' in __version__:
from pathlib import Path
Expand Down
7 changes: 4 additions & 3 deletions freqtrade/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,10 @@
'MaxDrawDownHyperOptLoss', 'MaxDrawDownRelativeHyperOptLoss',
'ProfitDrawDownHyperOptLoss']
AVAILABLE_PAIRLISTS = ['StaticPairList', 'VolumePairList', 'ProducerPairList', 'RemotePairList',
'AgeFilter', "FullTradesFilter", 'OffsetFilter', 'PerformanceFilter',
'PrecisionFilter', 'PriceFilter', 'RangeStabilityFilter',
'ShuffleFilter', 'SpreadFilter', 'VolatilityFilter']
'MarketCapPairList', 'AgeFilter', "FullTradesFilter", 'OffsetFilter',
'PerformanceFilter', 'PrecisionFilter', 'PriceFilter',
'RangeStabilityFilter', 'ShuffleFilter', 'SpreadFilter',
'VolatilityFilter']
AVAILABLE_PROTECTIONS = ['CooldownPeriod',
'LowProfitPairs', 'MaxDrawdown', 'StoplossGuard']
AVAILABLE_DATAHANDLERS = ['json', 'jsongz', 'hdf5', 'feather', 'parquet']
Expand Down
5 changes: 5 additions & 0 deletions freqtrade/persistence/trade_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,11 @@ def __repr__(self):
)

def to_json(self, minified: bool = False) -> Dict[str, Any]:
"""
:param minified: If True, only return a subset of the data is returned.
Only used for backtesting.
:return: Dictionary with trade data
"""
filled_or_open_orders = self.select_filled_or_open_orders()
orders_json = [order.to_json(self.entry_side, minified) for order in filled_or_open_orders]

Expand Down
157 changes: 157 additions & 0 deletions freqtrade/plugins/pairlist/MarketCapPairList.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
"""
Market Cap PairList provider
Provides dynamic pair list based on Market Cap
"""
import logging
from typing import Any, Dict, List

from cachetools import TTLCache
from pycoingecko import CoinGeckoAPI

from freqtrade.constants import Config
from freqtrade.exceptions import OperationalException
from freqtrade.exchange.types import Tickers
from freqtrade.plugins.pairlist.IPairList import IPairList, PairlistParameter


logger = logging.getLogger(__name__)


class MarketCapPairList(IPairList):

is_pairlist_generator = True

def __init__(self, exchange, pairlistmanager,
config: Config, pairlistconfig: Dict[str, Any],
pairlist_pos: int) -> None:
super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos)

if 'number_assets' not in self._pairlistconfig:
raise OperationalException(
'`number_assets` not specified. Please check your configuration '
'for "pairlist.config.number_assets"')

self._stake_currency = config['stake_currency']
self._number_assets = self._pairlistconfig['number_assets']
self._max_rank = self._pairlistconfig.get('max_rank', 30)
self._refresh_period = self._pairlistconfig.get('refresh_period', 86400)
self._marketcap_cache: TTLCache = TTLCache(maxsize=1, ttl=self._refresh_period)
self._def_candletype = self._config['candle_type_def']
self._coingekko: CoinGeckoAPI = CoinGeckoAPI()

if self._max_rank > 250:
raise OperationalException(
"This filter only support marketcap rank up to 250."
)

@property
def needstickers(self) -> bool:
"""
Boolean property defining if tickers are necessary.
If no Pairlist requires tickers, an empty Dict is passed
as tickers argument to filter_pairlist
"""
return False

def short_desc(self) -> str:
"""
Short whitelist method description - used for startup-messages
"""
num = self._number_assets
rank = self._max_rank
msg = f"{self.name} - {num} pairs placed within top {rank} market cap."
return msg

@staticmethod
def description() -> str:
return "Provides pair list based on CoinGecko's market cap rank."

@staticmethod
def available_parameters() -> Dict[str, PairlistParameter]:
return {
"number_assets": {
"type": "number",
"default": 30,
"description": "Number of assets",
"help": "Number of assets to use from the pairlist",
},
"max_rank": {
"type": "number",
"default": 30,
"description": "Max rank of assets",
"help": "Maximum rank of assets to use from the pairlist",
},
"refresh_period": {
"type": "number",
"default": 86400,
"description": "Refresh period",
"help": "Refresh period in seconds",
}
}

def gen_pairlist(self, tickers: Tickers) -> List[str]:
"""
Generate the pairlist
:param tickers: Tickers (from exchange.get_tickers). May be cached.
:return: List of pairs
"""
# Generate dynamic whitelist
# Must always run if this pairlist is the first in the list.
pairlist = self._marketcap_cache.get('pairlist_mc')
if pairlist:
# Item found - no refresh necessary
return pairlist.copy()
else:
# Use fresh pairlist
# Check if pair quote currency equals to the stake currency.
_pairlist = [k for k in self._exchange.get_markets(
quote_currencies=[self._stake_currency],
tradable_only=True, active_only=True).keys()]
# No point in testing for blacklisted pairs...
_pairlist = self.verify_blacklist(_pairlist, logger.info)

pairlist = self.filter_pairlist(_pairlist, tickers)
self._marketcap_cache['pairlist_mc'] = pairlist.copy()

return pairlist

def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
"""
Filters and sorts pairlist and returns the whitelist again.
Called on each bot iteration - please use internal caching if necessary
:param pairlist: pairlist to filter or sort
:param tickers: Tickers (from exchange.get_tickers). May be cached.
:return: new whitelist
"""
marketcap_list = self._marketcap_cache.get('marketcap')

if marketcap_list is None:
data = self._coingekko.get_coins_markets(vs_currency='usd', order='market_cap_desc',
per_page='250', page='1', sparkline='false',
locale='en')
if data:
marketcap_list = [row['symbol'] for row in data]
self._marketcap_cache['marketcap'] = marketcap_list

if marketcap_list:
filtered_pairlist = []

market = self._config['trading_mode']
pair_format = f"{self._stake_currency.upper()}"
if (market == 'futures'):
pair_format += f":{self._stake_currency.upper()}"

top_marketcap = marketcap_list[:self._max_rank:]

for mc_pair in top_marketcap:
test_pair = f"{mc_pair.upper()}/{pair_format}"
if test_pair in pairlist:
filtered_pairlist.append(test_pair)
if len(filtered_pairlist) == self._number_assets:
break

if len(filtered_pairlist) > 0:
return filtered_pairlist

return pairlist
Loading

0 comments on commit 22f2f9d

Please sign in to comment.