Are women more honest than men?

The journal Science recently published a fascinating article from Alain Cohn et al, which looked at cultural proclivities for honesty around the world via a rather ingenious method: they had research assistants turn in “lost” wallets all over the world and recorded when the receiver of the lost wallet attempted to return the wallet. The wallets were fake and included a false ID of a person who appeared to be local to the country in which the wallet was lost. The ingenious element was that instead of leaving the wallet in the open, the research assistants actually handed the wallet to somebody, thus enabling them to record interesting ancillary data like the age of the recipient, if they had a computer on their desk, and whether or not the person was local to the country. Clearly, the researchers were hoping to engage in a little bit of data mining to ensure their not insignificant efforts returned some publishable results regardless of the main outcome.

As it turns out, they needn’t have been concerned. The level of civic honesty, as measured by return rates, varied significantly. In addition, there is an interesting effect where the likelihood of the wallet being returned increases if there’s more money, an effect that persists across cultures. I encourage you to read the original article, which is fascinating, but a few of the highlights are that the US and Canada are somewhere in the middle, with about 55% return rates. On the top end of honesty are the Scandinavian and northern european countries, with rates at around 75%. On the bottom end of the curve are China and Peru, with about 14%. One could write an essay on the perils of letting the world economy become dependent on trade with a country with such a cultural bias against honesty, so I’ll leave that for another time. In the case of China, all the study did was confirm what anybody who does business in China knows: to them not cheating is a sign you’re not trying hard enough.

Here’s where things get really interesting: in keeping with modern scientific publishing standards, the researchers made their entire dataset available in an online data repository so that others could reproduce their work. There’s a lot of interesting conclusions one can make beyond what the authors pointed out in their paper, perhaps due to the political implications and the difficulty of doing a proper accounting for all the possible biases. But unburdened by the constraints of an academic career in the social sciences, I was more than happy to dig into the data to see what it could turn up…

Perhaps the most interesting thing I found is that women appear to be more honest than men. Over the entire world-wide dataset, women returned the wallets about 55%, versus 45% for men. This varied quite a bit over different cultures, and the data is not weighted by population, so one should take the absolute magnitude of this result with a bit of skepticism. However, looking at the data it appears that women were more honest than men in most cultures, so I think even a proper accounting for sample bias would find this result holds.

Women appear to be statistically significantly more honest than men. Standard error was less than 1% for both cases.

Here is the full dataset of men versus women broken down by country. You can see that the most populous countries are those where women appear to be more honest than men, so fixing the chart above to account for sample bias would likely still find a significant difference.

Women appear to be more honest than men in most cultures, though the individual country results are not usually statistically significant.

Another interesting question to ask of the data is whether or not there is a generational difference in honesty. Surprisingly, the answer turns out to be that there’s not a statistically significant difference:

Age doesn’t appear to be a statistically significant predictor of honesty. Standard error was roughly 1%, so the difference shown is not meaningful.

Looking at the breakdown by country, we see that there are no big differences between the generations, with one exception that I’m not even going to try to explain:

It’s possible that the young are more honest than the old, but it doesn’t appear to be statistically significant except in one country.

There are still a great deal of interesting variables in the dataset to explore. Please post a comment if you find any others.

Is technical analysis and algorithmic trading a legal form of market collusion?

Despite all dictates of logic against it, technical analysis actually works some of the time. I’m not talking about the random luck of any scheme working, but the fact that an uncanny amount of the time, “support” and “resistance” theories seem to be borne out in actual stock data. Assuming I’m not just imagining this, the obvious answer as to why is that it’s a self-fulfilling prophesy. If enough people believe in technical analysis, the patterns inherent to its doctrine will occur because the market is just composed of humans and their beliefs. It’s no more odd than the fact that stocks go up when earnings go up, except that belief at least has some basis in fact. The point is, having a basis in fact is irrelevant to the market.

Also obvious, however, is that when a stock pattern really works, it will draw attention to itself and it will stop working as people try to capitalize on it. Thus, while it may be a self-fulling prophesy that technical analysis will work, the same logic suggests that it should also fail, on average, because the market is mostly a zero-sum game. So, what gives?

It occurred to me that we are missing one vital factor: there are a lot of people who think technical analysis is utter bullshit. Ironically, if enough people think it’s bullshit, it just may work. It then dawned on me what technical analysis may really be and why it might actually work: a legal form of market collusion.

Technical analysis may work out to be a somewhat effective form of conspiracy among a subset of the market participants. The rules of TA are essentially a language, allowing adherents of the art to manipulate the stock market through mass distributed collusion. I think it is entirely possible that the “rules” of technical analysis are successful because they manipulate the market in a way that gives an advantage to adherents of those rules relative to those who don’t.

The net effect is that a group of people in the market cause prices to oscillate by all “colluding” to buy and sell at certain points. Another group of people then end up making trades at what the other half know to be the “wrong times” in a predictable cycle. The common rules of TA form a system of collusion that doesn’t require any back channel conversation between participants, and thus it is completely legal; the market itself is the channel through which the mass collusion occurs. The general disregard among academics for the possibility that TA could work results in a situation where there are always enough market participants who aren’t in on the game so that the game works. In fact, perhaps the best thing that ever happened for TA was the publication of “A Random Walk Down Wall Street.”

Free AppleScript for finding duplicates in the Apple Photos App on OS X

If you have a huge Photos library on the Mac, you very likely have a bunch of duplicate photos. People have also found that duplicates photos are sometimes created when you upgrade from iPhoto or Aperture to Photos. Unfortunately, if you look at the “free” programs on the App Store, you find that they are generally either scams or teasers for a paid version.

So, I wrote an AppleScript to go through every photo selected in Photos (you can select your entire library easily, if you want) and create a new Album with just the duplicate photos found. The script can be copied to your Photos script folder (if you have the script menu enabled) or you can just run it from with ScriptEditor. I also created a small script to reveal the ID of a selected photo, which can be useful for debugging the library and/or finding an original file within the inscrutable Photos database. Most people (if anybody) will just want the Find Duplicates script. Download them at the link below:

Photos AppleScripts

Once the script finds any duplicates, it creates a folder called “Duplicates” and adds a new album with the found duplicates. You should then go through that album and confirm that the duplicates are legitimate. If they are, select which the one duplicate you want to remove (you won’t want to get rid of the entire pair, presumably), right click and select delete. Make sure you don’t delete both in the pair, but if you do it’s not the end of the world; they will stay in the deleted photos album for a month.

Let me know if something doesn’t work right. The Photos AppleScript library doesn’t allow any destructive operations other than deleting albums (which I don’t do in either script) so the worst case should be that if it doesn’t work it just doesn’t find duplicates that it should.

Update: As a couple folks pointed out in the comments, the next step will be to create a script that allows you to “merge” a pair of photos, moving all the album associations and tags onto just one. I haven’t looked into whether this can even be accomplished with the Apple Script hooks available in Photos, so if anybody has an insight I’d appreciate it.

Verizon and Netflix problems resolved by routing around Cogent?

By now the problems between Verizon and Cogent are well known, at least to nerds. FiOS users, especially on the East Coast, have been complaining of increasingly poor Netflix streaming performance. The reason stems from the saturated connections between Cogent and Verizon which neither refuses to fix; see here for a good background on the dispute. Basically, Cogent is sending more data to Verizon than vice versa, and Verizon is asking Cogent to pay for the upgrades required. The reason this story has been getting any press is that it highlights the complete insanity of the commercial internet system: ISPs expect to get paid to carry data packets, even if those data packets are requested by their own customers. In this case, Cogent is simply providing Verizon with the Netflix data that Verizon’s customers have asked for.

To highlight the ludicrousness of the way the internet operates, Verizon could presumably generate traffic from Cogent for which it expects Cogent to pay by issuing requests to download data from Netflix itself. Or, as Netflix has pointed out, Netflix could resolve this situation by deciding to host its users backup data for them simply to artificially generate traffic going the other way. In fact, I’m surprised that Netflix doesn’t just program its streaming clients to repeat every bit back that they receive. That would solve this ludicrous problem, while also highlighting the stupidity of the way peering arrangements are made. At the bottom of this insanity is the fact that the companies who run networks have decided that they should get paid to carry packets like shipping companies would charge to carry packages. I would say it’s like UPS deciding to charge Amazon for shipping a package, while also deciding to charge the recipient for driving to their street. However, that’s not a perfect analogy, because if it were really like the Internet, UPS would be willing to waive the shipping if I handed them something to send back to Amazon. In fact, I struggle to find an analogy with the physical world of shipping, because there is no good analogy. Which is why it’s so incredibly stupid that network providers insist on billing arrangements that are analogous to shipping contracts.

Anyway, back to the point of this post: Comcast, which until recently had similar issues, has resolved them by getting Netflix to pay Comcast to connect directly to Netflix. There has been speculation Verizon would do the same. On the other hand, Verizon is probably not as willing to come to a reasonable solution as Comcast was, the latter trying to play nice to appease anti-trust regulators given it’s recent purchase of Time-Warner. I recently noticed an improvement in Netflix performance on FiOS, and wondered if maybe I was wrong about this. However, running a traceroute makes it clear that what happened is a third option I hadn’t considered; traffic between me and Netflix is going around Cogent and all the way to California:

3 g0-10-2-5.bstnma-lcr-21.verizon-gni.net (130.81.104.50)
4 ae1-0.bos-bb-rtr1.verizon-gni.net (130.81.151.60)
5 0.ae11.xl3.nyc1.alter.net (152.63.20.69)
6 0.xe-2-1-6.xt1.dca5.alter.net (152.63.0.113)
7 0.xe-4-1-3.xl3.iad8.alter.net (152.63.3.142)
8 tengige0-6-4-0.gw1.iad8.alter.net (152.63.35.145)
9 teliasonera-gw.customer.alter.net (152.179.50.234)
10 ash-bb4-link.telia.net (80.91.252.98)
11 las-bb1-link.telia.net (80.91.246.71)
12 netflix-ic-300871-las-bb1.c.telia.net (213.248.95.34)
13 ipv4_1.lagg0.c048.lax004.ix.nflxvideo.net (198.38.96.157)

Is it possible that rather than deal with Cogent or Netflix, Verizon has decided to just send East Coast Netflix traffic all the way to servers based in California, and to get there without using Cogent? Or, could Netflix have caused this by having clients make requests to different servers to get around the limited Cogent-Verizon ports? One thing that is clear is that we need a fundamentally different model for commercial internet if games like this are being played.

Stephen Wolfram is killing Mathematica

One of the saddest consequences of Steven Wolfram’s descent into megalomaniacal insanity (vis his decision to save science from itself by reinventing it in the image of a popular science book from the 1980s) is the continuing decline of Mathematica, his greatest (and, he seems intent upon forgetting, only) accomplishment.

Why the return to bitter posts? The week of my life I’ll never get back trying to get Mathematica‘s pitifully bad graph theory functions to yield correct results. I never thought I’d see the day when I considered MATLAB a superior product to Mathematica for doing something like network theory, but that day has come. I could go into great detail on the poor design of Mathematica‘s Graph object, but I’ll just leave the reader with the following object lesson on the perils of letting one’s ego interfere with one’s day job:

This is what happens when you decide to reinvent science but instead rediscover incompetence.

This is what happens when you decide to reinvent science but instead rediscover incompetence. (Note: the second graph has a lower “shortest path” despite losing an edge.)

Another nice bug is the fact that WeightedAdjacencyGraph[WeightedAdjacencyMatrix[g]] often returns an error, despite the obvious fact that it should return the original graph (at least topologically).

Seriously, Wolfram. Are there many more important mathematical topics today than graph theory? You can’t throw a copy of Mathematica these days (and I plan to) without hitting somebody working on a topic for which graph theory plays a central role. The fact that the interface to Graph[] is an unholy mess is nothing compared to the fact that it doesn’t even return correct results when things like GraphDistance[] are applied to a graph which has been manipulated. When Mathematica starts returning mathematically incorrect results, something is wrong with the world. That thing, I believe, is Stephen Wolfram himself. It’s time for him to move on from Wolfram and let somebody else run the show.

Looks like we’ll be here awhile…

I had originally made plans to move this to a private server, but since I’m working at MIT Lincoln Lab, it looks like I’m going to be able to keep all of my MIT computer access from when I was a graduate student. So, this will stay up indefinitely. I know you’re relieved. I may even start writing posts again…

How to rotate iPhone video on a Mac

I ran into what is apparently a not uncommon problem with iPhone video: you start to take a video while phone still thinks it’s in portrait orientation (long axis of the phone vertical) and then the rest of the video is stuck that way, even if you took 99% of it in landscape mode (the way God intended). You thus have a video that the Mac always wants to display sideways. If that didn’t make any sense, the bottom line is I had a video I needed to rotate 90 degrees, and maybe you do, too. While there were plenty of solutions available on the PC, tons of Googling turned up virtually nothing for the Mac, short of finding an old copy of iMovie from five years ago.

Fortunately, I lucked in to a great solution, which doesn’t even require transcoding (with the attendant loss in quality that would result). By the way, this worked for me in getting a video from portrait to landscape, and I suspect it may only work in that situation. (That said, this should be the only valid situation in which this problem occurs, as nobody in their right mind should ever shoot a video in portrait mode, and if you do, I’m certainly not going to be complicit in aiding and abetting that crime against humanity.)

The solution requires a copy of the very nice all-purpose video player, VLC.

  1. Open the video in VLC
  2. It should actually open up in landscape orientation, regardless of the erronious orientation data in the movie file from the iPhone.
  3. Select “Streaming/Exporting Wizard” from the File menu.
  4. Select “Transcode/Save to file” and click next.
  5. Use “Existing Playlist” and select the file you just opened below, click next.
  6. Leave everything untouched (i.e. both check boxes blank) on the transcode screen and click next.
  7. Choose MPEG4, click next.
  8. Click the button to tell VLC where to put the output file, then click next.
  9. Click finish.

This should be all it takes. The process will be fairly quick, since there’s no transcoding, but its not instantaneous as it does have to move a lot of bits into a new file.