I have a lot of experience with HFT. Lets clear all of this up.
How exchanges work for dummies.
All of the modern US stock exchanges work as a limit order book. Orders are filled on a price-time basis. In other words, you put in an order, if it isn't filled immediately it gets put in the limit order book with your order going to the end of the line ( lowest priority ) at that price. That's it, that is how it works. Lets look at an example.
I want to buy 250 shares of ABC. Right now there are orders in the book. 5 orders to buy at 9.95, each one for 100 shares. There are also 5 orders to sell at 9.96 also each order for 100 shares. I have several options.
(1) I can put in a market order to buy 250, when this order gets to the order book sell orders will be filled starting with the oldest ones, the ones that have been in the book longest, first. My order will be matched, this is why they call it a matching engine, with two of the sell orders and then a partial on the third one. As orders are filled in the order book messages are sent out to all subscribers of the data that those orders were filled so everyone can keep track of the state of the book. It is important to note that when my order gets to the matching engine/order book there is no chance for the orders in the order book to get cancelled. Those orders aren't told "you are about to get filled do you want to cancel".
(2) I can put in a limit order to buy at 9.96 in which case the order will be filled as described in 1 unless the sell orders at 9.96 are cancelled before my order gets to the matching engine.
(3) I can put in a limit order to buy at 9.95 in which case my order goes into the order book at 9.95 and I have to wait for the orders ahead of me to either get filled or cancelled and then I have to wait for someone to send in a sell order, either market or limit, that will get matched with my buy order.
That's it! Repeat, all day, every day. OK, that is a lie, there are special order types and that is one way exchanges try to differentiate themselves but that is a simplified way of how this works.
So where is the problem that Brad Katsuyama saw? I don't know him all I know is what I've seen and read. One thing to note is that he talks about buying large blocks of stock. Very large. From http://www.nytimes.com/2014/04/06/magazine/flash-boys-michae...
"It used to be that when his trading screens showed 10,000 shares of Intel offered at $22 a share, it meant that he could buy 10,000 shares of Intel for $22 a share." 10k shares at $22k/share, that is $220k in value. So he isn't talking about some small retail order, this is, as he points out, institutional flow.
So lets think about what happens in this kind of case. There are 13 exchanges, all running limit order books. They are geographically disperse and internally they are implemented differently and have different latencies and different message protocols etc. At some instance in time all 13 exchanges have some orders to sell at $22.00. Think about that. In some sense that is awesome. Look how efficient the market is. If you went to 13 gas stations what are the chances that a gallon of regular gas will cost exactly the same at every station? Anyway, back to our problem. You want to buy 10,000 shares. You fire off your order to exchange A to buy 10,000 shares. This hits the matching engine and you get filled on 10,000 shares ( THERE IS NO WAY FOR THOSE SELL ORDERS THAT ARE IN THE BOOK TO BE CANCELLED!!!! The orders in the book aren't sent a message "hey, a 10,000 share order is coming in and about to hit your order do you want to cancel your order first?". THAT DOES NOT HAPPEN!!! ). The thing is that those 10,000 shares that got matched, lets say each order in the book was 200 shares. Your 10,000 share order just took out 50 orders. The fact that these orders were executed is sent out to every data subscriber who is listening for that data. All messages are timestamped, sometimes down to the nanosecond. From that the recipients can tell "wow, all of these 50 orders for 10,000 shares were executed at the same time so that was one order."
Now, if you had orders in at other exchanges to sell at $22 what would your response be? For a second lets think about something more mundane, like TVs. Suppose you work at Target and you have a friend who works at Walmart. You're working one night and a guy shows up driving a semi-trailer. He walks in and says "How many 50 inch flat screens do you have? I'll buy them all, as advertised at $400." You sell all 20 that you have to him. As he's walking out the door he says "Now I'm going to Walmart." Just then your buddy at Walmart texts you and says "what's up? slow night right." and you go "Hell no, just sold 20 50 inch TVs to a guy, he bought everything I had and says he is headed for your Walmart to buy more." Now what does your friend do? In a world where prices are set by supply and demand Walmart could look at that and go "wow, there is a big buyer of 50 inch TVs out there, lets increase our price from $400 to $410." And so that is what Walmart does and sure enough the guy comes in and buys all the TVs from Walmart for $410. Does the TV buyer say the market is "rigged"?
In any case that is very similar to what happens in this case. All of the HFT and market makers see this big order rip through a lot of sell orders at exchange A and they go "wow, there is a big buyer, I better change my sell order from selling at $22.00 to selling at $22.01 or $22.10 or $25 or whatever". Brad Katsuyama looks at that and goes "I've been front run, this market is rigged." And then he has a piece of software written so that all orders arrive at the different markets at the same time, as opposed to leaving his trading workstation at the same time, and suddenly there isn't any front running. There isn't any front running at all, what he is seeing is the reaction to supply and demand.
HFT and market makers live in fear of something called "adverse selection". Market makers are required to post a two sided market, they are supposed to put up a price and quantity that they are willing to both buy and sell at right now. They publish this and they can't back away from it, they have to live up to it if another order shows up to buy or sell at a price that matches they have to execute the trade. The risk to them is that they are trading with someone who has information that they don't have. Just like playing poker or any game, you're going to lose if you play with people who are better than you. In this game though sometimes just having a big order is enough to move the market even if the person behind the order doesn't actually have any new information. Market makers and HFT either want to buy on the bid and sell on the offer and make the spread or they want to do arbitrage. That is how they make money. They lose money if they trade with people who are smarter or faster than they are. As soon as they see big orders come in they get really skittish and so they immediately move their orders to reduce their risk.
I've written a lot and bored almost everyone. Ask away and I'll do my best to answer your questions.
A couple other comments:
1. I'm not saying HFT is good or bad, I don't know.
2. Lots of people mention costs. Just to ballpark the whole thing. For the technology you need to Arista switches, that is $500/port, get a 48 port switch, that is $24k. For servers you need something decent, not crazy, dual CPU, 64 GB, running linux, maybe solarflare for kernel bypass networking, custom software. The hardware will cost you $6k/server. Then you need a connection to an exchange, varies but lets say $10k/month, then you need market data, $5k/month, then you need an order entry port, that is maybe $1k/month. That's it. Well, that is at one exchange, then you need low latency 10 Gbps cross connects between exchanges and you need to do that at each exchange. Then you need to find a clearing firm and you need to become a broker dealer ( exchanges and clearing firms only talk to BDs ). This is a highly regulated environment. You must have a compliance person, you will get audited yearly by SEC, your exchange, FINRA, all kinds of stuff. You will need a serious accountant. You have to have written procedures.
3. And how in the world do you avoid the Knight type of meltdown? Well, when you send in an order the order goes to the exchange. But the exchange also sends the order out via a "drop copy". This might go to your clearing firm so they know you aren't losing tons of money. But you can also get the drop copies and then you can constantly do your own risk checks that your orders you think are open are actually open but the ones you cancelled are actually cancelled. Most exchanges also have "cancel on disconnect" so when you really don't know what is going on you shut off your connection to the exchange and any open orders in the order book are cancelled for you.
How exchanges work for dummies.
All of the modern US stock exchanges work as a limit order book. Orders are filled on a price-time basis. In other words, you put in an order, if it isn't filled immediately it gets put in the limit order book with your order going to the end of the line ( lowest priority ) at that price. That's it, that is how it works. Lets look at an example.
I want to buy 250 shares of ABC. Right now there are orders in the book. 5 orders to buy at 9.95, each one for 100 shares. There are also 5 orders to sell at 9.96 also each order for 100 shares. I have several options.
(1) I can put in a market order to buy 250, when this order gets to the order book sell orders will be filled starting with the oldest ones, the ones that have been in the book longest, first. My order will be matched, this is why they call it a matching engine, with two of the sell orders and then a partial on the third one. As orders are filled in the order book messages are sent out to all subscribers of the data that those orders were filled so everyone can keep track of the state of the book. It is important to note that when my order gets to the matching engine/order book there is no chance for the orders in the order book to get cancelled. Those orders aren't told "you are about to get filled do you want to cancel". (2) I can put in a limit order to buy at 9.96 in which case the order will be filled as described in 1 unless the sell orders at 9.96 are cancelled before my order gets to the matching engine. (3) I can put in a limit order to buy at 9.95 in which case my order goes into the order book at 9.95 and I have to wait for the orders ahead of me to either get filled or cancelled and then I have to wait for someone to send in a sell order, either market or limit, that will get matched with my buy order.
That's it! Repeat, all day, every day. OK, that is a lie, there are special order types and that is one way exchanges try to differentiate themselves but that is a simplified way of how this works.
So where is the problem that Brad Katsuyama saw? I don't know him all I know is what I've seen and read. One thing to note is that he talks about buying large blocks of stock. Very large. From http://www.nytimes.com/2014/04/06/magazine/flash-boys-michae... "It used to be that when his trading screens showed 10,000 shares of Intel offered at $22 a share, it meant that he could buy 10,000 shares of Intel for $22 a share." 10k shares at $22k/share, that is $220k in value. So he isn't talking about some small retail order, this is, as he points out, institutional flow.
So lets think about what happens in this kind of case. There are 13 exchanges, all running limit order books. They are geographically disperse and internally they are implemented differently and have different latencies and different message protocols etc. At some instance in time all 13 exchanges have some orders to sell at $22.00. Think about that. In some sense that is awesome. Look how efficient the market is. If you went to 13 gas stations what are the chances that a gallon of regular gas will cost exactly the same at every station? Anyway, back to our problem. You want to buy 10,000 shares. You fire off your order to exchange A to buy 10,000 shares. This hits the matching engine and you get filled on 10,000 shares ( THERE IS NO WAY FOR THOSE SELL ORDERS THAT ARE IN THE BOOK TO BE CANCELLED!!!! The orders in the book aren't sent a message "hey, a 10,000 share order is coming in and about to hit your order do you want to cancel your order first?". THAT DOES NOT HAPPEN!!! ). The thing is that those 10,000 shares that got matched, lets say each order in the book was 200 shares. Your 10,000 share order just took out 50 orders. The fact that these orders were executed is sent out to every data subscriber who is listening for that data. All messages are timestamped, sometimes down to the nanosecond. From that the recipients can tell "wow, all of these 50 orders for 10,000 shares were executed at the same time so that was one order."
Now, if you had orders in at other exchanges to sell at $22 what would your response be? For a second lets think about something more mundane, like TVs. Suppose you work at Target and you have a friend who works at Walmart. You're working one night and a guy shows up driving a semi-trailer. He walks in and says "How many 50 inch flat screens do you have? I'll buy them all, as advertised at $400." You sell all 20 that you have to him. As he's walking out the door he says "Now I'm going to Walmart." Just then your buddy at Walmart texts you and says "what's up? slow night right." and you go "Hell no, just sold 20 50 inch TVs to a guy, he bought everything I had and says he is headed for your Walmart to buy more." Now what does your friend do? In a world where prices are set by supply and demand Walmart could look at that and go "wow, there is a big buyer of 50 inch TVs out there, lets increase our price from $400 to $410." And so that is what Walmart does and sure enough the guy comes in and buys all the TVs from Walmart for $410. Does the TV buyer say the market is "rigged"?
In any case that is very similar to what happens in this case. All of the HFT and market makers see this big order rip through a lot of sell orders at exchange A and they go "wow, there is a big buyer, I better change my sell order from selling at $22.00 to selling at $22.01 or $22.10 or $25 or whatever". Brad Katsuyama looks at that and goes "I've been front run, this market is rigged." And then he has a piece of software written so that all orders arrive at the different markets at the same time, as opposed to leaving his trading workstation at the same time, and suddenly there isn't any front running. There isn't any front running at all, what he is seeing is the reaction to supply and demand.
HFT and market makers live in fear of something called "adverse selection". Market makers are required to post a two sided market, they are supposed to put up a price and quantity that they are willing to both buy and sell at right now. They publish this and they can't back away from it, they have to live up to it if another order shows up to buy or sell at a price that matches they have to execute the trade. The risk to them is that they are trading with someone who has information that they don't have. Just like playing poker or any game, you're going to lose if you play with people who are better than you. In this game though sometimes just having a big order is enough to move the market even if the person behind the order doesn't actually have any new information. Market makers and HFT either want to buy on the bid and sell on the offer and make the spread or they want to do arbitrage. That is how they make money. They lose money if they trade with people who are smarter or faster than they are. As soon as they see big orders come in they get really skittish and so they immediately move their orders to reduce their risk.
I've written a lot and bored almost everyone. Ask away and I'll do my best to answer your questions.
A couple other comments:
1. I'm not saying HFT is good or bad, I don't know. 2. Lots of people mention costs. Just to ballpark the whole thing. For the technology you need to Arista switches, that is $500/port, get a 48 port switch, that is $24k. For servers you need something decent, not crazy, dual CPU, 64 GB, running linux, maybe solarflare for kernel bypass networking, custom software. The hardware will cost you $6k/server. Then you need a connection to an exchange, varies but lets say $10k/month, then you need market data, $5k/month, then you need an order entry port, that is maybe $1k/month. That's it. Well, that is at one exchange, then you need low latency 10 Gbps cross connects between exchanges and you need to do that at each exchange. Then you need to find a clearing firm and you need to become a broker dealer ( exchanges and clearing firms only talk to BDs ). This is a highly regulated environment. You must have a compliance person, you will get audited yearly by SEC, your exchange, FINRA, all kinds of stuff. You will need a serious accountant. You have to have written procedures. 3. And how in the world do you avoid the Knight type of meltdown? Well, when you send in an order the order goes to the exchange. But the exchange also sends the order out via a "drop copy". This might go to your clearing firm so they know you aren't losing tons of money. But you can also get the drop copies and then you can constantly do your own risk checks that your orders you think are open are actually open but the ones you cancelled are actually cancelled. Most exchanges also have "cancel on disconnect" so when you really don't know what is going on you shut off your connection to the exchange and any open orders in the order book are cancelled for you.