以下是使用MATLAB完成的。

我正在尝试使用深Q学习来构建交易算法。我刚刚花了一年的时间股价，并将其用作培训。

我的状态空间是我的 [money, stock, price]
money 是我拥有的现金数量，
stock 是我拥有的股票数量，
price 是当时股票的价格。

我遇到的问题是行动；在线寻找，人们只有三个动作， { buy | sell | hold }.

我的奖励功能是当前时间步骤中投资组合值的值与上一个时间步长之间的差异。

但是，只使用三个动作，我不确定如何选择购买，可以说67股股票？

我正在使用神经网络来近似Q值。它有三个输入 [money, stock, price] 和202个输出，即我可以出售0到100股数量，0，我可以持有股票，也可以购买1-100股股票。

任何人都可以阐明如何将其减少到3个动作？

我的代码是：

%  p is the stock price
% sp is the stock price at the next time interval 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

hidden_layers =   1;
actions       = 202;
net           = newff( [-1000000 1000000;-1000000 1000000;0 1000;],
                       [hidden_layers, actions],
                       {'tansig','purelin'},
                       'trainlm'
                       );

net           = init( net );

net.trainParam.showWindow = false;

% neural network training parameters -----------------------------------
net.trainParam.lr     =   0.01;
net.trainParam.mc     =   0.1;
net.trainParam.epochs = 100;

% parameters for q learning --------------------------------------------
epsilon        =    0.8;
gamma          =    0.95;
max_episodes   = 1000;
max_iterations = length( p ) - 1;

reset          =    false;
inital_money   = 1000;
inital_stock   =    0;

%These will be where I save the outputs
save_s        = zeros( max_iterations, max_episodes );
save_pt       = zeros( max_iterations, max_episodes );
save_Q_target = zeros( max_iterations, max_episodes );
save_a        = zeros( max_iterations, max_episodes );

% construct the inital state -------------------------------------------
% a           = randi( [1 3], 1, 1 );  
s             = [inital_money;inital_stock;p( 1, 1 )];


% construct initial q matrix -------------------------------------------
Qs            = zeros( 1, actions );
Qs_prime      = zeros( 1, actions );


for     i = 1:max_episodes
    for j = 1:max_iterations             % max_iterations --------------

        Qs = net( s );

        %% here we will choose an action based on epsilon-greedy strategy

        if ( rand() <= epsilon )
            [Qs_value  a] = max(Qs);
        else 
            a = randi( [1 202], 1, 1 );
        end

        a2                 = a - 101;
        save_a(j,i)        = a2;
        sp                 = p( j+1, 1 ) ; 
        pt                 = s( 1 ) + s( 2 ) * p( j, 1 );
        save_pt(j,i)       = pt; 
        [s_prime,reward]   = simulateStock( s, a2, pt, sp );

        Qs_prime           = net( s_prime );

        Q_target           = reward + gamma * max( Qs_prime );
        save_Q_target(j,i) = Q_target;
        Targets            = Qs;

        Targets( a )       =  Q_target;

        save_s( j, i )     = s( 1 );
        s                  = s_prime;
    end

    epsilon = epsilon * 0.99 ; 
    reset   = false; 
    s       = [inital_money;inital_stock;p(1,1)];
end

% ----------------------------------------------------------------------
function[s_prime,reward] = simulateStock( s, a, pt, sp )
                           money   = s(1);
                           stock   = s(2);
                           price   = s(3);

                           money   = money - a * price ;
                           money   = max( money, 0 );
                           stock   = s(2) + a;
                           stock   = max( stock, 0 );

                           s_prime = [money;stock;sp];
                           reward  = ( money + stock * price ) - pt;
end