There’s a huge issue in what you’re doing that need to have attention drawn to it.
I also created a weight variable which is the absolute value of the return, which I would use as input for xgboost.
If you have the information to calculate this, you already have the information to calculate the sign, as you don’t get to know any of the needed information until you observe the stock price. That is, you can’t predict tomorrow’s sign until you know that return, at which point, you know the sign of the return.
(From the comments, this seems to be used to weight the errors, rather than being an input feature that can’t be available when it comes time to make truly new predictions. I still have some issues with predicting signs of returns, but this is a clever way to partially allay some concerns of mine.)
The next point I’ll make is that ROCAUC has advantages over accuracy, but ROCAUC only deals with ranking. ROCAUC does not concern itself at all with how well the predicted probability values align with the reality of how often gains or losses really do happen. For that, common metrics are log loss and Brier score, each of which can be normalized as McFadden and Efron $R^2$, respectively.
Finally, I have seen others try to predict the signs of financial returns, seeming to take it as self-evident that being able to do so must mean trading riches, and seeing sign predictions instead of return predictions as being a way around markets being (mostly) efficient. From my perspective, you lose an enormous amount of valuable information by discarding the magnitude and only looking at the sign. For instance, when you make a trade based on the predicted sign, you incur a trading fee. If the return magnitudes are small, even if you correctly predict the sign, you could wind up losing money because the trading fees outweigh your returns (strong accuracy/ ROCAUC/Brier/etc yet you lose money). However, you will make mistakes in your sign predictions, and a run of gains due to good sign predictions can be offset by one big mistake. Your goal isn’t even to make money. You have to beat simple benchmarks like an S&P 500 index fund, and even if your returns are comparable, you have to consider the volatility in those returns and if it is worth incurring the uncertainty in how investing based on your model predictions.
Whether or not this will reliably work is not clear to me, but, philosophically, I think the right machine learning approach to investing comes from reinforcement learning: use the machine learning to create a strategy (the RL “policy”) that maximizes some kind of financial criterion like expected return or Sharpe ratio. This would combine both predictive modeling and what to do with the predictions. When you just predict the sign, you deny the trader such important information about what to do with that prediction, and naïvely buying/holding when signs are predicted to be positive and selling/staying out when signs are predicted to be negative is not necessarily a good strategy.
