When Apple moved to HEIC (a more efficient file format for photos) a few iOS versions ago (I can’t remember when, exactly) they created a very confusing situation where sometimes photos would be converted (if you had an arcane setting buried deep in iOS set a certain way) when you export them from your iPhone or Mac, and sometimes when you import from the iPhone to your Mac. And apparently Apple confused themselves, most of all, because many users (including myself) have reported mysterious double copies of photos cropping up in Apple Photos where a JPG version sits alongside the HEIC version. Obviously, nobody in their right minds wants this, so it appears to be some sort of bug.
Most people don’t care, but people with certain OCD tendencies–which is probably the target demographic of Apple, to be honest–are really bugged by this. So, in the spirit of mental health I wrote an AppleScript to find any photo where there is both a JPG and an HEIC version with the same file name, and put the JPG version into a special album for you to review and potentially delete. You can find the script here:
To run the script from inside Script Editor is fairly simple. Select a bunch of photos in Photos (you probably don’t want to try to run through groups of more than 2500 at a time) and then run the script. It will take a minute or so and will then create an album of the duplicates. If you end up having a bunch, Photos may ask you to confirm you actually want to do this (so, if the script appears hung, it may be Photos waiting for your input). Good luck!
The journal Science recently published a fascinating article from Alain Cohn et al, which looked at cultural proclivities for civic honesty around the globe. They employed a rather ingenious method: they “lost” wallets all over the world and recorded when the receiver of the lost wallet attempted to return the wallet to its rightful owner. The wallets were fake and included a false ID of a person who appeared to be local to the country in which the wallet was lost, including fake contact info that actually belonged to the researchers. The ingenious element of the research was that instead of leaving the wallet out in the open, the research assistants actually pretended to have found the wallets in our nearby local businesses and turned in the wallet to somebody working in that business, thus enabling them to record interesting ancillary data on the “subject,” such as their age, if they had a computer on their desk, and whether or not the person was local to the country. Clearly, the researchers were hoping to engage in a little bit of data mining to ensure their not insignificant efforts returned some publishable results regardless of the main outcome.
As it turns out, they needn’t have been concerned. The level of civic honesty, as measured by wallet return rates, varied significantly between cultures. In addition, there is an interesting effect where the likelihood of the wallet being returned increased if there was more money in it, an effect that persists across regions and which was evidently not predicted by most economists. I encourage you to read the original article, which is fascinating. On the top end of the civic honesty scale are the Scandinavian and Northern European countries, with rates at around 75%. On the bottom end of the curve is China, with about 14%. In the case of China, all the study did was confirm what anybody who does business there knows, and something that has been well covered by journalists and completely ignored by our politicians: to the Chinese, not cheating is a sign you’re not trying hard enough.
Here’s where things get interesting: in keeping with modern scientific publishing standards, the researchers made their entire dataset available in an online data repository so that others could reproduce their work. There are a lot of interesting conclusions one can make beyond what the authors were willing to point out in their paper, perhaps due to the political implications and the difficulty of doing a proper accounting for all the possible biases. However, unburdened by the constraints of an academic career in the social sciences, I was more than happy to dig into the data to see what it could turn up…
Perhaps the most interesting thing I found is that women appear to be more honest than men. Over the entire world-wide dataset, women returned the wallets about 51% of the time, versus 42% for men. It is tempting to look at individual countries, but the male versus female difference is not statistically significant enough when looking at individual countries, so I chose to only look at the aggregate data. The data is not weighted by country population, so one should take the absolute magnitude of the difference with a bit of skepticism. However, looking at the individual country data it appears a proper accounting for population bias would likely maintain or increase the difference. (Some of the most populous countries had the largest difference between women and men.)
Here is the full dataset of men versus women broken down by country. You can see that the most populous countries are those where women appear to be more honest than men, so fixing the chart above to account for sample bias would likely still find a significant difference.
Another interesting question to ask of the data is whether or not there is a generational difference in honesty. Surprisingly, the answer turns out to be that there’s not a statistically significant difference:
Looking at the breakdown by country, we see that there are no big differences between the generations, with one exception that I’m not even going to try to explain:
One interesting set of issues that always comes up with population studies like this is what, if anything, should we do with this information? It is true that a Swedish woman is about eight times more civically honest, on average, than a Chinese man. That’s interesting, but also pretty dangerous information. Should this inform our immigration policy, where population statistics might actually be valid? Is it better to not even ask these questions given the abuse of the information that might result? Or, is it good to have this information, especially when it flies in the face of our image of ourselves and others? I suspect in the case of the US, most would be surprised to find out that the average US citizen is as honest as the average Russian. We may be surprised by both halves of that statement, and both might be good to think about.
Despite all dictates of logic against it, technical analysis actually works some of the time. I’m not talking about the random luck of any scheme working, but the fact that an uncanny amount of the time, “support” and “resistance” theories seem to be borne out in actual stock data. Assuming I’m not just imagining this, the obvious answer as to why is that it’s a self-fulfilling prophesy. If enough people believe in technical analysis, the patterns inherent to its doctrine will occur because the market is just composed of humans and their beliefs. It’s no more odd than the fact that stocks go up when earnings go up, except that belief at least has some basis in fact. The point is, having a basis in fact is irrelevant to the market.
Also obvious, however, is that when a stock pattern really works, it will draw attention to itself and it will stop working as people try to capitalize on it. Thus, while it may be a self-fulling prophesy that technical analysis will work, the same logic suggests that it should also fail, on average, because the market is mostly a zero-sum game. So, what gives?
It occurred to me that we are missing one vital factor: there are a lot of people who think technical analysis is utter bullshit. Ironically, if enough people think it’s bullshit, it just may work. It then dawned on me what technical analysis may really be and why it might actually work: a legal form of market collusion.
Technical analysis may work out to be a somewhat effective form of conspiracy among a subset of the market participants. The rules of TA are essentially a language, allowing adherents of the art to manipulate the stock market through mass distributed collusion. I think it is entirely possible that the “rules” of technical analysis are successful because they manipulate the market in a way that gives an advantage to adherents of those rules relative to those who don’t.
The net effect is that a group of people in the market cause prices to oscillate by all “colluding” to buy and sell at certain points. Another group of people then end up making trades at what the other half know to be the “wrong times” in a predictable cycle. The common rules of TA form a system of collusion that doesn’t require any back channel conversation between participants, and thus it is completely legal; the market itself is the channel through which the mass collusion occurs. The general disregard among academics for the possibility that TA could work results in a situation where there are always enough market participants who aren’t in on the game so that the game works. In fact, perhaps the best thing that ever happened for TA was the publication of “A Random Walk Down Wall Street.”
If you have a huge Photos library on the Mac, you very likely have a bunch of duplicate photos. People have also found that duplicates photos are sometimes created when you upgrade from iPhoto or Aperture to Photos. Unfortunately, if you look at the “free” programs on the App Store, you find that they are generally either scams or teasers for a paid version.
So, I wrote an AppleScript to go through every photo selected in Photos and create a new Album with just the duplicate photos found. (With a large album I recommend selecting one or two years at a time to avoid issues with memories.) The script can be copied to your Photos script folder (if you have the script menu enabled) or you can just run it from with ScriptEditor. I also created a small script to reveal the ID of a selected photo, which can be useful for debugging the library and/or finding an original file within the inscrutable Photos database. Most people (if anybody) will just want the Find Duplicates script. Download them at the link below:
Once the script finds any duplicates, it creates a folder called “Duplicates” and adds a new album with the found duplicates. You should then go through that album and confirm that the duplicates are legitimate. If they are, select which the one duplicate you want to remove (you won’t want to get rid of the entire pair, presumably), right click and select delete. Make sure you don’t delete both in the pair, but if you do it’s not the end of the world; they will stay in the deleted photos album for a month.
Let me know if something doesn’t work right. The Photos AppleScript library doesn’t allow any destructive operations other than deleting albums (which I don’t do in either script) so the worst case should be that if it doesn’t work it just doesn’t find duplicates that it should.
Update: As a couple folks pointed out in the comments, the next step will be to create a script that allows you to “merge” a pair of photos, moving all the album associations and tags onto just one. I haven’t looked into whether this can even be accomplished with the AppleScript hooks available in Photos, so if anybody has an insight I’d appreciate it.
By now the problems between Verizon and Cogent are well known, at least to nerds. FiOS users, especially on the East Coast, have been complaining of increasingly poor Netflix streaming performance. The reason stems from the saturated connections between Cogent and Verizon which neither refuses to fix; see here for a good background on the dispute. Basically, Cogent is sending more data to Verizon than vice versa, and Verizon is asking Cogent to pay for the upgrades required. The reason this story has been getting any press is that it highlights the complete insanity of the commercial internet system: ISPs expect to get paid to carry data packets, even if those data packets are requested by their own customers. In this case, Cogent is simply providing Verizon with the Netflix data that Verizon’s customers have asked for.
To highlight the ludicrousness of the way the internet operates, Verizon could presumably generate traffic from Cogent for which it expects Cogent to pay by issuing requests to download data from Netflix itself. Or, as Netflix has pointed out, Netflix could resolve this situation by deciding to host its users backup data for them simply to artificially generate traffic going the other way. In fact, I’m surprised that Netflix doesn’t just program its streaming clients to repeat every bit back that they receive. That would solve this ludicrous problem, while also highlighting the stupidity of the way peering arrangements are made. At the bottom of this insanity is the fact that the companies who run networks have decided that they should get paid to carry packets like shipping companies would charge to carry packages. I would say it’s like UPS deciding to charge Amazon for shipping a package, while also deciding to charge the recipient for driving to their street. However, that’s not a perfect analogy, because if it were really like the Internet, UPS would be willing to waive the shipping if I handed them something to send back to Amazon. In fact, I struggle to find an analogy with the physical world of shipping, because there is no good analogy. Which is why it’s so incredibly stupid that network providers insist on billing arrangements that are analogous to shipping contracts.
Anyway, back to the point of this post: Comcast, which until recently had similar issues, has resolved them by getting Netflix to pay Comcast to connect directly to Netflix. There has been speculation Verizon would do the same. On the other hand, Verizon is probably not as willing to come to a reasonable solution as Comcast was, the latter trying to play nice to appease anti-trust regulators given it’s recent purchase of Time-Warner. I recently noticed an improvement in Netflix performance on FiOS, and wondered if maybe I was wrong about this. However, running a traceroute makes it clear that what happened is a third option I hadn’t considered; traffic between me and Netflix is going around Cogent and all the way to California:
Is it possible that rather than deal with Cogent or Netflix, Verizon has decided to just send East Coast Netflix traffic all the way to servers based in California, and to get there without using Cogent? Or, could Netflix have caused this by having clients make requests to different servers to get around the limited Cogent-Verizon ports? One thing that is clear is that we need a fundamentally different model for commercial internet if games like this are being played.
One of the saddest consequences of Steven Wolfram’s descent into megalomaniacal insanity (vis his decision to save science from itself by reinventing it in the image of a popular science book from the 1980s) is the continuing decline of Mathematica, his greatest (and, he seems intent upon forgetting, only) accomplishment.
Why the return to bitter posts? The week of my life I’ll never get back trying to get Mathematica‘s pitifully bad graph theory functions to yield correct results. I never thought I’d see the day when I considered MATLAB a superior product to Mathematica for doing something like network theory, but that day has come. I could go into great detail on the poor design of Mathematica‘s Graph object, but I’ll just leave the reader with the following object lesson on the perils of letting one’s ego interfere with one’s day job:
This is what happens when you decide to reinvent science but instead rediscover incompetence. (Note: the second graph has a lower “shortest path” despite losing an edge.)
Another nice bug is the fact that WeightedAdjacencyGraph[WeightedAdjacencyMatrix[g]] often returns an error, despite the obvious fact that it should return the original graph (at least topologically).
Seriously, Wolfram. Are there many more important mathematical topics today than graph theory? You can’t throw a copy of Mathematica these days (and I plan to) without hitting somebody working on a topic for which graph theory plays a central role. The fact that the interface to Graph is an unholy mess is nothing compared to the fact that it doesn’t even return correct results when things like GraphDistance are applied to a graph which has been manipulated. When Mathematica starts returning mathematically incorrect results, something is wrong with the world. That thing, I believe, is Stephen Wolfram himself. It’s time for him to move on from Wolfram and let somebody else run the show.
I had originally made plans to move this to a private server, but since I’m working at MIT Lincoln Lab, it looks like I’m going to be able to keep all of my MIT computer access from when I was a graduate student. So, this will stay up indefinitely. I know you’re relieved. I may even start writing posts again…