What we do is try to recreate the name we have in front of us. Neither device is totally accurate and the garbage ends up in the “small counts”. These devices let us “skip out of the real universe” ie it lets us create non-names and we have no basis after the fact to reject many of these phantoms.

There is a solution: it is to load the known names (the finite universe) into the desktop, laptop, hand held devices, et cetera, and not allow any name that is not known to the machine to enter any system until it has been reviewed. If it is really a new-to-the system name, the sytem gets updated with the name.

We need to take name capture from a ‘recreate’ exercise, which sometimes, unfortunately becomes a ‘create’ execise, to a simple ‘look-up’

Ken

Nevertheless, I don’t think he has resolved the mystery in this case. Here are the raw and cumulative name frequencies, reorganized into bins of uniform size:

freq                  raw                cum
1+                5493558            6248415
10+                603186             754857
100+               128015             151671
1000+               20369              23656
10000+               3012               3287
100000+               268                275
1000000+                7                  7

If you take logs and fit a linear function, you’ll find there’s only a tiny difference in the slope—and the change is in the wrong direction. For the cumulative numbers, the slope is 0.942; for the raw numbers 0.929. (Given that both ends of this curve look a little fishy, it’s probably better to fit only to the middle five points. The slopes in that case are 0.854 and 0.833.)

Of course the possibility that I have made some misteak remains very much alive.

Is the discrepancy not explained by the fact that the MDZ exponent is for the number of clans with *exactly* m members, while your exponent is, as you point out, for the cumulative statistic of the number of clans with m members or more? Roughly speaking, your power law is the integral of 1/x^2 (the MDZ power law) from m to infinity, which is 1/m. Or am I failing, as usual, to understand some subtle, obvious point?

@Jess: It’s surely true that the geographic distribution of names is not i.i.d., but if we can’t solve the problem in that simple case, we’re going to have trouble with more realistic and more complicated distributions.

The claim that name frequencies follow a power law with exponent β = 2 is in fact based on an empirical observation. (There may be some theoretical justification as well.) Manrubia et al. graph name data from earlier U.S. Census reports and from the Berlin phone book, and in both cases the slope (judged by eyeball) strongly suggests β = 2. Why the new Census data should give such a different result is perplexing. Of course it’s quite possible that I made some blunder in my own analysis.

You’re assuming that samples within a geographic area are independent variables. That is demonstrably not the case here: it wouldn’t be strange, for example, for all five holders of a given surname in the U.S. to live in the same house as a family.

This question calls to mind the distinction that N. N. Taleb draws between the “Gaussian” and the scalable. Since the researchers have demonstrated a nice power-law distribution, we’re definitely in scalable territory. I don’t know what argument the researchers made for the exponent being 2. I think if you could get a consistent non-2 estimate from several different U.S. cities, you’d have just as strong an argument for that exponent.

It was studied from a computer science perspective in the recent paper:
Sofya Raskhodnikova, Dana Ron, Amir Shpilka and Adam Smith.
Strong Lower Bounds for Approximating Distribution Support Size and the Distinct Elements Problem (FOCS 2007).
I haven’t read this paper, though, so don’t know how practically relevant it really is.

I think it is clear that your assumptions will make a big difference. For example, consider two distributions of American surnames:
a. 300 million different surnames (all unique),
b. 150 million different surnames, each appearing twice (none unique).
Clearly, you can’t tell these two distributions apart unless your random sample is large enough to see a collision (which takes a sample of size sqrt{300 million} = 17000).