Tomorrow I’ll be doing a gig on “The State of Things,” broadcast by North Carolina Public Radio. The main subject of discussion will be Scott Huler’s new book On the Grid; I’ll be present as supporting cast (call me a bit player) and will doubtless find a way to plug my own book Infrastructure.
If you’re within listening range of a North Carolina Public Radio station, tune in between noon and 1 p.m. Later in the day the audio should be available online.
Martin Gardner died over the weekend. He was 95 and living in Norman, Oklahoma, not too far from his birthplace in Tulsa.
Like many others, I grew up on Martin’s “Mathematical Games” column in Scientific American. Later I joined the staff of that magazine—but don’t imagine that Martin and I became office buddies. As a matter of fact, I never once saw him in the office. He worked at home. He attended none of our editorial meetings. We never had lunch together. Indeed, I never would have met him at all except for the coincidence that we lived a few blocks apart, and every now and then I would be called upon to deliver a package of urgent proofs. (By the way, his address in those days was on Euclid Avenue!)
Someone on the magazine staff described Martin as “a shy woodland creature,” and the tag stuck. Looking back, however, I think it tells only half the story. Yes, Martin was no schmoozer, and he preferred to stay out of the spotlight, both in person and in print. Most of his best-known columns reported on someone else’s discoveries—Conway’s game of life, RSA’s cryptosystem, Penrose’s tilings. He delighted in annotating other people’s work, as in his celebrated edition of Lewis Carroll. Yet he was anything but timid or retiring. In an argument, the shy woodland creature was a grizzly bear. He had strongly held opinions and philosophical convictions, and he knew the worth of his own work. He was a man of ideas, to be taken seriously, yet also a man who had fun with his ideas.
As a celebration of Martin’s peculiar genius, I would like to revive the little puzzle that formed the basis of his very first column, in January of 1957. (He had published a few articles in Scientific American earlier, but this was the first column to appear under the “Mathematical Games” title.)
On the bingo card above, choose any number, circle it, and cross out all the other numbers in the same column or row. Now select a second number from among those that remain unmarked, and again circle your choice and then cross out the rest of the column and row. Continue in this way until there are no unmarked numbers left to choose.
The sum of the circled numbers is 57. How did I know that? How did Martin construct the matrix?
Update 2010-05-29: Here is Martin’s own explanation of the 1957 puzzle:
Like most tricks, this one is absurdly simple when explained. The square is nothing more than an old-fashioned addition table, arranged in a tricky way. The table is generated by two sets of numbers: 12, 1, 4, 18, 0 and 7, 0, 4, 9, 2. The sum of these numbers is 57. If you write the first set of numbers horizontally above the top row of the square, and the second set vertically beside the first column [see Fig. 9], you can see at once how the numbers in the cells are determined. The number in the first cell (top row, first column) is the sum of 12 and 7, and so on through the square.
You can construct a magic square of this kind as large as you like and with any combination of numbers you choose. It does not matter in the least how many cells the square contains or what numbers are used for generating it. They may be positive or negative, integers or fractions, rationals or irrationals. The resulting table will always possess the magic property of forcing a number by the procedure described, and this number will always be the sum of the two sets of numbers that generate the table.
If you were an astute or lucky stock trader on the afternoon of May 6, you could have bought shares of Accenture PLC for a penny each and sold them a minute later for almost $40. Or you could have invested in Sotheby’s for about $30 a share and, if your timing was right, sold out at a price of $99,999.9999. Did you miss those moneymaking opportunities? Don’t kick yourself too hard. Those particular trades were canceled by the exchanges as “clearly erroneous errors.” But millions of other bizarre transactions were allowed to stand, even though prices were fluctuating wildly.
A preliminary report on these events was released last week by a joint committee of the Commodity Futures Trading Commission and the Securities and Exchange Commission. The report reads a lot like an inquiry into an airplane crash, evoking both horror and fascination. But whereas the investigators of aircraft accidents usually come up with a likely cause, the CFTC/SEC committee makes clear that they don’t yet understand what happened on May 6, and it seems possible we’ll never know.
Throughout that day, stock prices were trending lower, a decline attributed mainly to worries about the European economy. But those concerns can’t account for the extraordinary crevasse the market fell into and then climbed out of between 2:30 and 3:00 p.m. The Dow Jones Industrial Average (blue) and the Standard and Poor’s 500 index (green) both lost 6 or 7 percent of their value in less than 10 minutes, then gained it all back. If those price changes are extrapolated to all U.S. stocks, something like a trillion dollars went missing for half an hour. (The red line in the graph, labeled E-Mini S&P 500, refers to a stock futures contract, which I’ll discuss below.)
What could cause such rapid whipsawing? The first speculations implicated a “fat-finger trade”—a data-entry error. There have been several such events in recent years; for example, in 2005 a Japanese broker who meant to sell 1 share of stock at a price of 610,000 yen keyed in instructions to sell 610,000 shares at 1 yen. However, the committee finds no evidence of such goofs on May 6.
The committee also dismisses the Procter & Gamble theory, put forward by commentators on CNBC who noticed a particularly sharp break in the stock of that company (one of the 30 Dow components).
The decline in PG did not begin until 2:44 p.m., well after the broader market indices, which began their precipitous drop at approximately 2:40 p.m. Accordingly, early reports that an inordinately large trade in PG may have triggered the broad market decline do not appear well founded.
Various kinds of deliberate mischief have also been mentioned as possible causes. Maybe some secretive hedge fund has found a way to manipulate the market to its own advantage. Or a hacker might have infiltrated the computer networks that handle stock transactions. The glitch could even be an act of international terrorism. Again, the committee finds no signs of such malevolence but can’t entirely rule out the possibility.
The committee gives closer scrutiny to high-volume trading on the stock futures market, and in particular to the E-Mini S&P 500 futures, which offer a mechanism for betting on the value of the S&P 500 index a few weeks in the future. Traffic in S&P 500 futures was unusually heavy on May 6, and it spiked at the time of the big dip:
The price excursions were wide enough to trigger a “Stop Logic” system that halted trading for five seconds. Furthermore, transactions initiated by a single firm accounted for some 9 percent of the trading volume in the critical half-hour, and all of that firm’s activity was on the selling side. (The committee report does not name this firm, but others have identified it as Waddell & Reed, a mutual fund in Overland Park, Kansas.) So, do we blame it all on a mutual fund run amok in the KC suburbs? The committee thinks further investigation is warranted, but they also note that the same firm has made similar trades in the past, as have many other parties, all without causing a ripple in the wider market.
Two more items of Wall Street arcana that get a lot of attention in the report are stop-loss orders and stub quotes. A stop-loss order causes a stock to be sold automatically if the price falls below a specified threshold. Traders enter such orders in the expectation that the sale will take place at a price near the threshold level, but if prices are falling rapidly, there’s no assurance of that. For a few minutes on May 6, certain stop-loss orders had the effect not of stopping losses but of maximizing them. At the instant when the orders were executed, there were no purchase offers at any price higher than a penny, and so that’s the price the stocks sold for. The offers of $0.01 are thought to have been “stub quotes,” placed by brokers who act as market-makers and who are therefore obliged always to have both buy and sell orders in place. Stub quotes are a way of meeting this obligation at times when the broker doesn’t really want to be in the market. Trades are never supposed to be executed at the stub price, but that’s what happens if no one else is buying. (Transactions at $100,000 per share reflect stub quotes at the other end of the scale, for shares that no one else is willing to sell.)
• • •
If the Commodity Futures Trading Commission and the Securities and Exchange Commission don’t know what went wrong on May 6, then I’m sure I don’t know either. But a couple of points seem pretty obvious (which may be why the committee left them unstated).
First, whatever happened on May 6 must have been driven by the internal dynamics of the securities markets, not by events in the larger economy. No changes in the business prospects of Accenture PLC would justify 4,000 percent swings in the company’s market value within half an hour.
Second, there’s got to be some instability at work here—some positive feedback loop. A thousand-point dip in the Dow wasn’t just a freak coincidence, where millions of stockholders acting independently all chose to sell at the same moment, and then a few minutes later changed their minds and decided to buy. Rather, there must have been some mechanism whereby one trader’s decision to buy or sell induced other traders to do the same.
The committee report points out that stop-loss orders create one such destabilizing loop, which is hard-wired into the market machinery. If a stop-loss order on a particular stock is activated at $100, say, the sale of those shares might drive the market price down to $95, triggering more stop-loss orders and lowering the price still further, in a runaway cascade. More generally, any trading strategy that calls for following trends or tracking “market momentum” is susceptible to this kind of instability. For any one individual, selling out when the market sags may or may not be a prudent policy; but if everyone adopts such a rule, the outcome is certain disaster.
Positive feedbacks of some kind surely had a role in the crash of May 6, but they can’t be the whole story. If a wave of self-reinforcing selling accounts for the sudden dive in prices, what explains the equally sudden turnaround and recovery? And there’s an even deeper question. It’s not hard to dream up models in which every random fluctuation is amplified by positive feedback, but the result is an economy that experiences weird jolts and hiccoughs all the time. A useful theory of May 6 has to explain not only what happened on that day but also why it doesn’t happen routinely.
Some analysts have compared the May 6 event with the stock market crash of October 1987, which was even deeper than the recent dip, although it played out over a period of days rather than minutes. I have vivid memories of this event; I followed it on the radio (no CNBC in those days) and then I read the post-mortem reports. But apparently my memory is faulty in certain crucial details. The crash was blamed in large part on “program trading,” which I took to mean that computer programs were making buy and sell decisions in real time. The root of the problem, as I understood it then, was that multiple programs controlling large investments all shared the same basic logic, so that they would all react in the same way to changing market conditions. It turns out, though, that the computing machinery of the time was not up to operating in this online regime. Instead, the economic models were run in batch mode, and the trades were executed after the fact. There were people in the loop.
Today, in contrast, thousands of computers are plugged directly into the markets, and program trading is everywhere. The big hedge funds and other major players install their servers in colocation facilities next door to the major exchanges, as a way of reducing communication latency. For “high frequency traders,” transactions are routinely completed in about a third of a millisecond. From the point of view of these firms, the sudden market collapse on May 6 played out in slow motion. During the 10 minutes of tumbling prices, a trading rate of three transactions per millisecond allows time for 180,000 transactions.
Perhaps, then, the much-feared runaway automation of 1987 has finally caught up with us in 2010. Ironically, though, the CFTC/SEC report hints that if automated trading was behind the May 6 glitch, the problem might not be the presence of these traders but rather their sudden withdrawal from the market. Julie Creswell tells the story in The New York Times:
RED BANK, N.J. — Above the Restoration Hardware in this Jersey Shore town, not far from the Navesink River, lurks a Wall Street giant.
Here, inside the humdrum offices of a tiny trading firm called Tradeworx, workers in their 20s and 30s in jeans and T-shirts quietly tend high-speed computers that typically buy and sell 80 million shares a day.
But on the afternoon of May 6, as the stock market began to plunge in the “flash crash,” someone here walked up to one of those computers and typed the command HF STOP: sell everything, and shutdown.
According to Creswell, high-frequency traders account for between 40 and 70 percent of all the trading volume on U.S. securities markets, so the sudden departure of these market participants would certainly have a noticeable effect.
Almost everything about the stock market has changed utterly in the years since 1987. Back then, trading was done by guys in colorful blazers yelling at one another on the floor of the New York Stock Exchange. That trading floor still exists, but it’s a kind of Wall Street theme park, maintained for the benefit of visiting high school classes and CNBC cameras. Most of the actual trading in NYSE stocks is done across the river in Jersey City by electronic ”matching engines” that line up offers to sell with bids to buy. Once there were “specialists” in each stock who were expect to intervene with their own capital to damp out unwarranted price fluctuations. That role has not disappeared entirely, but in most modern markets no one has legal responsibility for maintaining stability. In 1987 most stocks could be bought and sold in only one venue; now, transactions are automatically routed to whatever exchange offers the best terms, including the ominously named “dark pools,” where shares change hands anonymously. Back then, brokerage fees and other transaction costs were high enough to discourage strategies such as high-frequency trading; now there is much less friction in the market. It’s a new world.
Even though the CFTC and the SEC have not yet sorted out the causes of the May 6 blip, they are already proposing remedies. The basic tool is the time out: When the market throws a tantrum, it will be told to sit in the corner for a few minutes. Many such rules already exist, some of them going back to 1987. The rationale is that a pause in trading will allow time for “additional liquidity to enter the market.” In other words, if everyone is selling in a panic, we wait a little while for some buyers to show up. Of course the pause might also allow time for more sellers to join the stampede.
A year ago, I was writing about the uneasy relations between economics and the engineering discipline known as control theory. That was in the context of macroeconomics, where the aim is to control cycles of boom and bust with a time scale of years or decades. The challenges of controlling securities markets are rather different: The time scale is much shorter, which means you have to act quicker, but on the other hand it’s much easier to measure what’s happening, to gather information second by second. But the biggest impediment to effective control is the same in both cases: It’s hard to control the dynamics of a system when you don’t understand those dynamics—when you can’t reliably predict what the system will do in the absence of control or how it will respond to control actions. Given the human element in economic affairs—including the likely presence of actors who will try to subvert any control strategy—it’s not clear that we can ever have that kind of predictive power.
The Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables (better known as Abramowitz and Stegun) is a much-storied book. Not that it’s a book full of stories; truth is, there’s not much of a narrative thread running through those formulas, graphs and mathematical tables. But it’s a book with a story behind it.
The story began in the late 1930s, when the Mathematical Tables Project was launched in a factory building on the West Side of Manhattan. Supported by the Works Progress Administration, the Tables Project had dual aims: first, preparing high-quality tables of trigonometric functions, logarithms, and the like; and, second, providing work for unemployed New Yorkers. The 450 human computers hired for the project were chosen more on the basis of need than skill, and the work was done on a kind of numerical assembly line. According to David Alan Grier:
Each group was taught to perform a single arithmetic operation. One group knew how to add positive numbers, a second to subtract, and the third to multiply single digits. The last and most sophisticated group did long division.
I have to admit there’s a certain nightmare aspect to this scene. Working in a numbers factory sounds no more appealing than stamping sheet metal all day, although there was less danger of getting a finger crushed in the machinery.
A few years later, the Tables Project was swept up in war work; then, afterward, many of the key personnel moved to the National Bureau of Standards (now NIST, the National Institute of Standards and Technology). There they conceived the Handbook. Apparently the initial plan was a greatest-hits album of tables, but by the early 1950s the future of table-making was looking pretty dim. And so the project changed course and put more emphasis on the mathematical functions that lay behind the tables—familiar functions such logarithms, more specialized ones such as Bessel functions and some more recondite topics such as Mathieu functions and orthogonal polynomials. There were still tables listing numeric values of functions, but the Handbook also presented the mathematics you would need to evaluate (or approximate) the function for yourself.
The editors in charge of the Handbook were Milton Abramowitz and Irene Stegun, both veterans of the New York table office. They recruited about 30 young mathematicians to write chapters. In 1958 Abramowitz died suddenly; Stegun saw the project through to publication in 1964.
The Handbook seems an unlikely best-seller, but the U.S. government has distributed more than 150,000 copies, and editions from other publishers are estimated to bring the total copies in print to something near a million. (As a product of government work, the Handbook is not covered by copyright; there are scanned versions on the web.)
Announced last week is a new Handbook, officially retitled the NIST Handbook of Mathematical Functions. The ink-and-paper version, which I have not yet seen, is published by Cambridge University Press. Perhaps even more interesting is the web edition, called the NIST Digital Library of Mathematical Functions (DLMF), which I have just begun to explore. It is recognizably the same book as Abramowitz and Stegun, with the same terse style of presentation. But much has changed. The hundreds of pages of tables are finally gone; this is not the place to look up the sine of 23 degrees. But there are handsome color graphics now, and a new emphasis on methods of computation, including pointers to recommended software. And the selection of topics has expanded somewhat. For example, there are new chapters on the Painlevé equations and on functions whose argument is a matrix. Elsewhere, the Lambert W function (a personal favorite of mine) is a newcomer to the chapter on elementary functions.
Apart from the content, the DLMF is interesting as an experiment in presenting mathematics on the web. It’s the most ambitious project I’ve seen based on MathML, and it seems to work well, at least when viewed in recent versions of Firefox. (In other browsers I’ve tried, MathML gets garbled, but the equations can be displayed as images and are still quite readable in that format—even with an ancient version of Internet Explorer.) Here’s part of a page as seen in Firefox:
Mousing over the “i” icon at the right margin provides access to encodings of the equations in MathML or TeX and as PNG images, as well as definitions, cross-references and such. Very slick.
The editor in chief of the new Handbook is Frank W. J. Olver of the University of Maryland College Park and NIST, whom I have mentioned before both here at bit-player and in American Scientist. As a young mathematician, half a century ago, Olver wrote one of the Handbook chapters on Bessel functions. (As an even younger mathematician, in the 1940s, he worked with Alan Turing at the National Physical Laboratory in Britain.) There are three more principal editors: Daniel W. Lozier, Ronald F. Boisvert and Charles W. Clark, all of NIST, as well as a long roster of associate editors and domain experts. It’s too soon to say whether some combination of these names will eventually replace the moniker “Abramowitz and Stegun.”
Notes: The quotation above from David Alan Grier appears in “The Math Tables Project of the Work Projects Administration: The Reluctant Start of the Computing Era,” IEEE Annals of the History of Computing, Vol. 20, No. 3, 1998. Grier has also written a profile of Stegun in “Irene Stegun, the Handbook of Mathematical Functions, and the Lingering Influence of the New Deal,” American Mathematical Monthly, August-September 2006. Boisvert and Lozier have written a brief account of the history of the Handbook. (Oddly, the images in this PDF file are negatives.)
In the past few weeks I’ve had little time for bit-playing; I’ve been playing with atoms instead. I’ve been sorting and packing and toting atoms, then hauling them, rearranging them, offloading them, storing them. Lots and lots of atoms: maybe 1029. Bits are so much easier to handle. My bitly possessions—some hundreds of gigabytes—fit comfortably in a shirt pocket. My atomic chattels are a bulkier burden.
All this atom-pushing was done in the course of moving my household from Durham, North Carolina, to Cambridge, Massachusetts. I’ve moved before, but this time the experience was unusually physical. In vacating my Durham home, I carried all my belongings out of the house and onto a truck—and I did it without helpers and without the use of carts or dollies or other wheeled implements. In other words, everything I own (except a car) I have now lifted up, cradled in my arms, carried 50 feet or more, and set down again. It took a day and a half.
What a goofy thing to do, eh? I even turned away offers of help. I guess I’m just a do-it-yourself kind of guy. In a strange way the labor was worth it: The process gave me a vivid and visceral sense of how stuff accumulates over a lifetime—how my possessions possess me. Late in the afternoon of the second day, as I loaded up the last few items and shut the door of the truck, there was a bright glow of satisfaction and accomplishment.
Yet I never want to do it again. When I arrived in Cambridge, I did not refuse the generous help of a younger, stronger friend. (Thanks, Mici!)
The total weight of my load was roughly two and a half metric tons. At least two tons of that mass consisted of “information goods”—books, periodicals, manuscripts and proofs, file drawers full of paper documents, photographs, musical recordings in various formats, art works. And this was the residue remaining after a two-year effort to lighten my load, mainly by transforming atoms into bits. In particular, I had scanned 22 drawers full of files, converting paper into PDFs and then recycling all the cellulose.
Before I lug my belongings out of this dwelling, I vow to jettison another ton or more. If only I could figure out how to digitize my clothes or my pots and pans.
As for my new home, everyone knows that Cambridge is the intellectual capital of North America. But I didn’t quite realize how high the standard had become. The sign in the photo below is on the garden gate next door.
The literate local canines seem to comply with this order, since the gate is always closed.