If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage. About the only thing slower are USB, floppy drives and tape.
So basically big systems only make sense if the computation time on a single system exceeds the transfer time off of that system to the remote nodes + the time for that data to work it's way up the local node memory hierarchy.
This can happen when the data is big and the computation has unavoidable serialism in each part of the computation. Otherwise you'r likely to be able to coax a single system to just do it faster and with far less complexity than working on a compute cluster.
(I'm ignoring big systems with fast shared memory, like classic super-computers, where there effectively is zero transfer time since each node uses the same memory pool)
> If you think of the network as part of the memory hierarchy, transferring data to nodes in a distributed system is usually slower than even spinning disk storage.
According to Google quick search, the bandwidth of DDR3 is ~12.8 GB/s, or ~104 gbps. As I understand it second-hand (read: cheap) infiniband givies ~40 gpbs, and newer, high-end gives ~100gbps[1]. Now, if you're using regular gigabit ethernet (or even 10gps with cheap-ish switches) -- you'd probably see around 5-8 gbps -- or if we're being very generous, on the order of 1/10th of cpu-ram bandwidth.
Sata2 is rated at ~6 gbps for comparison. In other words, 10gpbs ethernet/infiniband should be on-par with local SSDs, but "proper" infiniband should push the node-node ram-ram performance up compared to that. It appears fiberchannel is also moving towards the ~100 gps mark[2], but isn't there quite yet. And you'd have to have something on the other end -- with high speed networking it is quite natural to assume that both source and destination is RAM.
It's actually pretty crazy with ~40gps infiniband -- I'm considering getting some second-hand gear just to play with at home. Seems both cheaper and more reliable to work with than try to get 10gps ethernet to work (on a home/hobby budget!).
All that said, I think the article makes a lot of great points -- and it does seem a bit funny that people seemingly are thinking of 1 gbps as "fast" when so much has happened on the cpu/ram side since that was remotely true.
Though, you'd be surprised at how often people get stuck on distributed problems where they just can't seem to get their cluster to process more than 80-100MB/sec and it turns out it's because they ran 1Gb ethernet between their nodes.
If I'm reading the numbers right[1], and converting correctly, infiniband is on the order of 5000 - 500 ns. I didn't mean to imply that the network isn't slower, just that it's possible to do quite a lot better than gigabit ethernet quite cheaply.
Yup. Infiniband + RDMA is pretty cool, but RDMA is also rather scary. I mean, writing remotely to another systems (physical) memory. In the same time I do wonder why this is not being used more nowadays.
So basically big systems only make sense if the computation time on a single system exceeds the transfer time off of that system to the remote nodes + the time for that data to work it's way up the local node memory hierarchy.
This can happen when the data is big and the computation has unavoidable serialism in each part of the computation. Otherwise you'r likely to be able to coax a single system to just do it faster and with far less complexity than working on a compute cluster.
(I'm ignoring big systems with fast shared memory, like classic super-computers, where there effectively is zero transfer time since each node uses the same memory pool)