Memory leak problem in Ichimoku Indicator.

[Superior18]

Memory usage will increase until it reaches 100%.
This is happening slowly. For example, it will increase by 15% every 10 minutes.
This problem causes the robot to be automatically killed by the operating system after about 45 minutes.
My strategy :

# --- Do not remove these libs ---
from freqtrade.strategy.interface import IStrategy
from typing import Dict, List
from functools import reduce
from pandas import DataFrame
# --------------------------------
import technical.indicators as ta


class icu(IStrategy):
    """

    author@: Gert Wohlgemuth

    idea:

        uptrend definition:
            MACD above MACD signal
            and CCI < -50

        downtrend definition:
            MACD below MACD signal
            and CCI > 100

    """

    # Minimal ROI designed for the strategy.
    # This attribute will be overridden if the config file contains "minimal_roi"
    minimal_roi = {
        "120":  0.03,
        "60":  0.06,
        "40":  0.08,
        "20":  0.1
    }

    # Optimal stoploss designed for the strategy
    # This attribute will be overridden if the config file contains "stoploss"
    stoploss = -0.1
    order_types = {
    "buy": "limit",
    "sell": "limit",
    "emergencysell": "market",
    "stoploss": "market",
    "stoploss_on_exchange": False,
    "stoploss_on_exchange_interval": 60,
    "stoploss_on_exchange_limit_ratio": 0.99,
    }
    # Optimal ticker interval for the strategy
    ticker_interval = '5m'

    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        # ichimoku
        ichi = ta.ichimoku(dataframe)
        dataframe['tenkan']=ichi['tenkan_sen']
        dataframe['kijun']=ichi['kijun_sen']
        dataframe['senkou_a']=ichi['senkou_span_a']
        dataframe['senkou_b']=ichi['senkou_span_b']
        dataframe['cloud_green']=ichi['cloud_green']
        dataframe['cloud_red']=ichi['cloud_red']

        return dataframe

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe.loc[
            (
                (dataframe['tenkan'].shift(1)<dataframe['kijun'].shift(1)) &
                (dataframe['tenkan']>dataframe['kijun'])
                (dataframe['cloud_red']==True)
            ),
            'buy'] = 1
        return dataframe

    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        dataframe.loc[
            (
            ),
            'sell'] = 1
        return dataframe

[xmatthias]

there's not much information in this issue to be honest... (not enough to even start to help you, anyway)

First of all, which version of freqtrade are you using? on which operating system, and with which version of Python?

Also, what's the total memory you've got on your system?
What command are you running (trade, backtesting, hyperopt, ...)?

[Superior18]

freqtrade Version : freqtrade develop-39ea126
python version : Python 3.8.2 (also tested with python 3.6.9 and get same result in memory usage)
OS : Debian 8 32 bit version.
physical system memory 2 GB.
freqtrade command : freqtrade trade
and also.
I set up my strategy with FinTA and it works.
With the technical module, increase memory usage to infinity occurs.

[xmatthias]

just to understand this right - by only using ichimoku (as shown in the above sample), you're experiencing a memory leak ...

but when using ichimoku from FinTA it's fine... ?

[Superior18]

Yes. Exactly so.
It seems that in the technical module, the Ichimoku indicator has a problem and creates a memory leak.

[xmatthias]

Am i correct in assuming that the missing ampersand in line 67 (2nd line of the buy condition) is just a typo and in reality you're not seeing any errors on the console?

Apart from that - finta seems to have ichimoku wrong.

I've only looked at tenkan - which according to investopedia and python for finance is calculated as (Period high + period low) / 2 - which to me, is (df['high'].rolling(period).max() + df['low'].rolling(period).max()) / 2.
Finta uses .mean() instead of max and min - which will result in different values.

Link to finta source:

https://github.com/peerchemist/finta/blob/30b491e7301351b45d61d059a43deedb7dcd0a3b/finta/finta.py#L1538-L1545

---

Due to finta not using .min() and .max() - they're not hitting pandas-dev/pandas#32266 - a pandas bug which seems to have been introduced in 1.0.0 (and is still present in 1.0.3).

I've tried this with the previous pandas version (0.25.3) - and indeed, memory usage seems stable.
pip install -U pandas==0.25.3

As a workaround, best downgrade pandas (we're not using anything new in panas 1.0 for now).

[xmatthias]

Since pandas 1.0.4 was released, this should no longer be a problem.

It's now a dependency for both freqtrade (master / develop) and for the technical repository.