Market Making via Reinforcement Learning

ABSTRACT

Market making is a fundamental trading problem in which an agent provides liquidity by continually offering to buy and sell a security. The problem is challenging due to inventory risk, the risk of accumulating an unfavourable position and ultimately losing money. In this paper, we develop a high-fidelity simulation of limit order book markets, and use it to design a market making agent using temporal-difference reinforcement learning. We use a linear combination of tile codings as a value function approximator, and design a custom reward function that controls inventory risk. We demonstrate the effectiveness of our approach by showing that our agent outperforms both simple benchmark strategies and a recent online learning approach from the literature.

References

1. Frederic Abergel, Anirban Chakraborti, Aymen Jedidi, Ioane Muni Toke, and Marouane Anane . 2016. Limit Order Books. Cambridge University Press.
2. Jacob Abernethy and Satyen Kale . 2013. Adaptive Market Making via Online Learning. In Proc. of NIPS. 2058--2066.
3. Marco Avellaneda and Sasha Stoikov . 2008. High-frequency trading in a limit order book. Quantitative Finance, Vol. 8, 3 (2008), 217--224.
4. Pierre-Luc Bacon, Jean Harb, and Doina Precup . 2016. The Option-Critic Architecture. CoRR Vol. abs/1609.05140 (2016).
5. Marc G Bellemare, Will Dabney, and Rémi Munos . 2017. A Distributional Perspective on Reinforcement Learning International Conference on Machine Learning. 449--458.
6. Aseem Brahma, Mithun Chakraborty, Sanmay Das, Allen Lavoie, and Malik Magdon-Ismail . 2012. A Bayesian market maker. In Proc. of EC. New York, New York, USA, 215--232.
7. Álvaro Cartea and Sebastian Jaimungal . 2015. Risk Metrics and Fine Tuning of High-Frequency Trading Strategies. Mathematical Finance, Vol. 25, 3 (2015), 576--611.
8. Álvaro Cartea, Sebastian Jaimungal, and Damir Kinzebulatov . 2016. Algorithmic Trading with Learning. International Journal of Theoretical and Applied Finance, Vol. 19, 04 (2016), 1650028.
9. Álvaro Cartea, Sebastian Jaimungal, and José Penalva . 2015. Algorithmic and High-Frequency Trading. Cambridge University Press.
10. Álvaro Cartea, Sebastian Jaimungal, and Jason Ricci . 2014. Buy Low Sell High: A High Frequency Trading Perspective. SIAM Journal on Financial Mathematics Vol. 5, 1 (2014), 415--444.
11. Tanmoy Chakraborty and Michael Kearns . 2011. Market Making and Mean Reversion. In Proc. of EC. 307--314.
12. Nicholas T. Chan and Christian R. Shelton . 2001. An Electronic Market-Maker. AI Memo 2001-005. MIT AI Lab.
13. Hugh L Christensen, Richard E Turner, Simon I Hill, and Simon J Godsill . 2013. Rebuilding the limit order book: sequential Bayesian inference on hidden states. Quantitative Finance, Vol. 13, 11 (2013), 1779--1799.
14. Dave Cliff . 2006. ZIP60: an enhanced variant of the ZIP trading algorithm CEC/EEE.
15. Kristopher De Asis, J Fernando Hernandez-Garcia, G Zacharias Holland, and Richard S Sutton . 2017. Multi-step reinforcement learning: A unifying algorithm. arXiv preprint arXiv:1703.01327 (2017).
16. M. A H Dempster and V. Leemans . 2006. An automated FX trading system using adaptive reinforcement learning. Expert Systems with Applications Vol. 30, 3 (2006), 543--552.
17. Pei-yong Duan and Hui-he Shao . 1999. Multiple Hyperball CMAC Structure for Large Dimension Mapping. Proc. of IFAC, Vol. 32, 2 (1999), 5237--5242.
18. Dhananjay K. Gode and Shyam Sunder . 1993. Allocative Efficiency of Markets with Zero-Intelligence Traders: Market as a Partial Substitute for Individual Rationality. Journal of Political Economy Vol. 101, 1 (1993), 119--137.
19. Martin D Gould, Mason A Porter, Stacy Williams, Mark McDonald, Daniel J Fenn, and Sam D. Howison . 2013. Limit order books. Quantitative Finance, Vol. 13, 11 (2013), 1709--1742.
20. Sanford J Grossman and Merton H Miller . 1988. Liquidity and Market Structure. The Journal of Finance Vol. 43, 3 (1988), 617.
21. Marek Grzes and Daniel Kudenko . 2010. Reward Shaping and Mixed Resolution Function Approximation. Developments in Intelligent Agent Technologies and Multi-Agent Systems. IGI Global, Chapter 7.
22. Olivier Guéant, Charles Albert Lehalle, and Joaquin Fernandez-Tapia . 2013. Dealing with the inventory risk: A solution to the market making problem. Mathematics and Financial Economics Vol. 7, 4 (2013), 477--507.
23. Fabien Guilbaud and Huyen Pham . 2011. Optimal High Frequency Trading with limit and market orders. CoRR Vol. abs/1106.5040 (2011).
24. Seijen Harm van, Hasselt Hado van, Shimon Whiteson, and Marco Wiering . 2009. A theoretical and empirical analysis of expected SARSA Proc. of ADPRL. 177--184.
25. Joel Hasbrouck and Gideon Saar . 2013. Low-latency trading. Journal of Financial Markets Vol. 16, 4 (2013), 646--679.
26. Hado V Hasselt . 2010. Double Q-learning Proc. of NIPS. 2613--2621.
27. Richard Haynes and John S Roberts . 2015. Automated Trading in Futures Markets. CFTC White Paper (2015).
28. Thomas Ho and Hans R. Stoll . 1981. Optimal dealer pricing under transactions and return uncertainty. Journal of Financial Economics Vol. 9, 1 (1981), 47--73.
29. M. Leaver and T. W. Reader . 2016. Human Factors in Financial Trading: An Analysis of Trading Incidents. Human Factors, Vol. 58, 6 (2016), 814--832.
30. Hamid Reza Maei, Csaba Szepesvari, Shalabh Bhatnagar, and Richard Sutton . 2010. Toward Off-Policy Learning Control with Function Approximation. Proc. of ICML, 719--726.
31. Warwick Masson and George Konidaris . 2016. Reinforcement Learning with Parameterized Actions. Proc. of AAAI, 1934--1940.
32. John E Moody and Matthew Saffell . 1998. Reinforcement Learning for Trading. In Proc. of NIPS. 917--923.
33. Yuriy Nevmyvaka, Yi Feng, and Michael Kearns . 2006. Reinforcement learning for optimized trade execution Proc. of ICML. 673--680.
34. Abraham Othman . 2012. Automated Market Making: Theory and Practice. Ph.D. Dissertation. bibinfoschoolCarnegie Mellon University.
35. Abraham Othman, David M Pennock, Daniel M Reeves, and Tuomas Sandholm . 2013. A Practical Liquidity-Sensitive Automated Market Maker. ACM Transactions on Economics and Computation, Vol. 1, 3 (2013), 1--25.
36. Rahul Savani . 2012. High-frequency trading: The faster, the better? IEEE Intelligent Systems Vol. 27, 4 (2012), 70--73.
37. L. Julian Schvartzman and Michael P. Wellman . 2009. Stronger CDA Strategies through Empirical Game-Theoretic Analysis and Reinforcement Learning. Proc. of AAMAS (2009), 249--256.
38. C R Shelton . 2001. Importance Sampling for Reinforcement Learning with Multiple Objectives. Ph.D. Dissertation. bibinfoschoolMassachusetts Institute of Technology.
39. Alexander A. Sherstov and Peter Stone . 2004. Three Automated Stock-Trading Agents: A Comparative Study. Agent Mediated Electronic Commerce VI: Theories for and Engineering of Distributed Mechanisms and Systems (AMEC 2004). Vol. Vol. 3435. 173--187.
40. Tom Spooner, John Fearnley, Rahul Savani, and Andreas Koukorinis . 2018. Market Making via Reinforcement Learning. (2018). showeprint{arxiv}1804.04216
41. R.S. Sutton and A.G. Barto . 1998. Reinforcement Learning: An Introduction. IEEE Transactions on Neural Networks Vol. 9, 5 (1998), 1054--1054.
42. Harm Van Seijen, A Rupam Mahmood, Patrick M Pilarski, Marlos C Machado, and Richard S Sutton . 2016. True online temporal-difference learning. Journal of Machine Learning Research Vol. 17, 145 (2016), 1--40.
43. Perukrishnen Vytelingum, Dave Cliff, and Nicholas R Jennings . 2008. Strategic bidding in continuous double auctions. Artif. Intell., Vol. 172, 14 (2008), 1700--1729.

Index Terms

• Applied computing

  • Law, social and behavioral sciences

    • Economics

• Computing methodologies

  • Machine learning

    • Learning paradigms

      • Reinforcement learning

  • Modeling and simulation