Free AppleScript for removing duplicates in Apple Photos App

If you have a huge photo library on the Mac, you very likely have a bunch of duplicate photos. People have also found that duplicates photos are sometimes created when you upgrade from iPhoto or Aperture to Photos. Unfortunately, if you look at the “free” programs on the App Store, you find that they are generally either scams or teasers for a paid version.

So, I wrote an AppleScript to go through every photo selected in Photos (you can select your entire library easily, if you want) and create a new Album with just the duplicate photos found. The script can be copied to your Photos script folder (if you have the script menu enabled) or you can just run it from with ScriptEditor. I also created a small script to reveal the ID of a selected photo, which can be useful for debugging the library and/or finding an original file within the inscrutable Photos database. Most people (if anybody) will just want the Find Duplicates script. Download them at the link below:

Photos AppleScripts

Once the script finds any duplicates, it creates a folder called “Duplicates” and adds a new album with the found duplicates. You should then go through that album and confirm that the duplicates are legitimate. If they are, select which the one duplicate you want to remove (you won’t want to get rid of the entire pair, presumably), right click and select delete. Make sure you don’t delete both in the pair, but if you do it’s not the end of the world; they will stay in the deleted photos album for a month.

Let me know if something doesn’t work right. The Photos AppleScript library doesn’t allow any destructive operations other than deleting albums (which I don’t do in either script) so the worst case should be that if it doesn’t work it just doesn’t find duplicates that it should.

Verizon and Netflix problems resolved by routing around Cogent?

By now the problems between Verizon and Cogent are well known, at least to nerds. FiOS users, especially on the East Coast, have been complaining of increasingly poor Netflix streaming performance. The reason stems from the saturated connections between Cogent and Verizon which neither refuses to fix; see here for a good background on the dispute. Basically, Cogent is sending more data to Verizon than vice versa, and Verizon is asking Cogent to pay for the upgrades required. The reason this story has been getting any press is that it highlights the complete insanity of the commercial internet system: ISPs expect to get paid to carry data packets, even if those data packets are requested by their own customers. In this case, Cogent is simply providing Verizon with the Netflix data that Verizon’s customers have asked for.

To highlight the ludicrousness of the way the internet operates, Verizon could presumably generate traffic from Cogent for which it expects Cogent to pay by issuing requests to download data from Netflix itself. Or, as Netflix has pointed out, Netflix could resolve this situation by deciding to host its users backup data for them simply to artificially generate traffic going the other way. In fact, I’m surprised that Netflix doesn’t just program its streaming clients to repeat every bit back that they receive. That would solve this ludicrous problem, while also highlighting the stupidity of the way peering arrangements are made. At the bottom of this insanity is the fact that the companies who run networks have decided that they should get paid to carry packets like shipping companies would charge to carry packages. I would say it’s like UPS deciding to charge Amazon for shipping a package, while also deciding to charge the recipient for driving to their street. However, that’s not a perfect analogy, because if it were really like the Internet, UPS would be willing to waive the shipping if I handed them something to send back to Amazon. In fact, I struggle to find an analogy with the physical world of shipping, because there is no good analogy. Which is why it’s so incredibly stupid that network providers insist on billing arrangements that are analogous to shipping contracts.

Anyway, back to the point of this post: Comcast, which until recently had similar issues, has resolved them by getting Netflix to pay Comcast to connect directly to Netflix. There has been speculation Verizon would do the same. On the other hand, Verizon is probably not as willing to come to a reasonable solution as Comcast was, the latter trying to play nice to appease anti-trust regulators given it’s recent purchase of Time-Warner. I recently noticed an improvement in Netflix performance on FiOS, and wondered if maybe I was wrong about this. However, running a traceroute makes it clear that what happened is a third option I hadn’t considered; traffic between me and Netflix is going around Cogent and all the way to California:

3 (
4 (
5 (
6 (
7 (
8 (
9 (
10 (
11 (
12 (
13 (

Is it possible that rather than deal with Cogent or Netflix, Verizon has decided to just send East Coast Netflix traffic all the way to servers based in California, and to get there without using Cogent? Or, could Netflix have caused this by having clients make requests to different servers to get around the limited Cogent-Verizon ports? One thing that is clear is that we need a fundamentally different model for commercial internet if games like this are being played.

Stephen Wolfram is killing Mathematica

One of the saddest consequences of Steven Wolfram’s descent into megalomaniacal insanity (vis his decision to save science from itself by reinventing it in the image of a popular science book from the 1980s) is the continuing decline of Mathematica, his greatest (and, he seems intent upon forgetting, only) accomplishment.

Why the return to bitter posts? The week of my life I’ll never get back trying to get Mathematica‘s pitifully bad graph theory functions to yield correct results. I never thought I’d see the day when I considered MATLAB a superior product to Mathematica for doing something like network theory, but that day has come. I could go into great detail on the poor design of Mathematica‘s Graph object, but I’ll just leave the reader with the following object lesson on the perils of letting one’s ego interfere with one’s day job:

This is what happens when you decide to reinvent science but instead rediscover incompetence.

This is what happens when you decide to reinvent science but instead rediscover incompetence. (Note: the second graph has a lower “shortest path” despite losing an edge.)

Another nice bug is the fact that WeightedAdjacencyGraph[WeightedAdjacencyMatrix[g]] often returns an error, despite the obvious fact that it should return the original graph (at least topologically).

Seriously, Wolfram. Are there many more important mathematical topics today than graph theory? You can’t throw a copy of Mathematica these days (and I plan to) without hitting somebody working on a topic for which graph theory plays a central role. The fact that the interface to Graph[] is an unholy mess is nothing compared to the fact that it doesn’t even return correct results when things like GraphDistance[] are applied to a graph which has been manipulated. When Mathematica starts returning mathematically incorrect results, something is wrong with the world. That thing, I believe, is Stephen Wolfram himself. It’s time for him to move on from Wolfram and let somebody else run the show.

Looks like we’ll be here awhile…

I had originally made plans to move this to a private server, but since I’m working at MIT Lincoln Lab, it looks like I’m going to be able to keep all of my MIT computer access from when I was a graduate student. So, this will stay up indefinitely. I know you’re relieved. I may even start writing posts again…

How to rotate iPhone video on a Mac

I ran into what is apparently a not uncommon problem with iPhone video: you start to take a video while the camera is in portrait mode (just by accident of how you’re holding it) and then the rest of the video is stuck that way, even if you took 99% of it in landscape mode. If that didn’t make any sense, the bottom line is I had a video I needed to rotate 90 degrees. While there were plenty of solutions available on the PC, tons of Googling turned up virtually nothing for Mac Os X, short of finding an old copy of iMovie from five years ago.

Fortunately, I lucked in to a great solution, which doesn’t even require transcoding (with the attendant loss in quality that would result). This worked for me in getting a video from portrait to landscape, and I suspect it will only work in that situation. (This should be the only situation in which this problem occurs, as nobody in their right mind should ever shoot video in portrait mode on purpose, and if you do, I’m certainly not going to be complicit in aiding and abetting that crime against humanity.)

The solution requires a copy of the very nice all-purpose video player, VLC.

  1. Open the video in VLC
  2. It should actually open up in landscape orientation, regardless of the erronious orientation data in the movie file from the iPhone.
  3. Select “Streaming/Exporting Wizard” from the File menu.
  4. Select “Transcode/Save to file” and click next.
  5. Use “Existing Playlist” and select the file you just opened below, click next.
  6. Leave everything untouched (i.e. both check boxes blank) on the transcode screen and click next.
  7. Choose MPEG4, click next.
  8. Click “Choose…” to tell VLC where to put the output file, click next.
  9. Click finish.

This should be all it takes. The process will be fairly quick, since there’s no transcoding, but its not instantaneous as it does have to move a lot of bits into a new file.

Proof you’re a parent

For some reason I decided to take a look at the top 50 most played songs on my iPod. Here they are, listed from 5 to 1.

Rank Title               Artist             Number of Times Played
5.   Many the Miles      Sara Bareilles     66
4.   Fireflies           Owl City           67
3.   The Wider Sun       Jon Hopkins        81
2.   Baby Monkey         Parry Gripp        322
1.   Oatmeal             Parry Gripp        393

One guess as to which two songs are Alex’s favorites?

Hypothesis testing proves ESP is real

Or maybe it would be more accurate to say that ESP proves frequentist statistics isn’t real. In what has got to be one of the best object lessons in why hypothesis testing (the same statistical method usually used by the medical research industry to produce the Scare of the Week) is prone to generate false results, a well-respected psychology journal is set to publish a paper describing a statistically significant finding that ESP works.

The real news here, of course, is not that ESP has been proven real, but that using statistics to try to understand the world is a breeding ground for junk science. If I had to guess, I’d say the author (who is a well-respected academic who has never published any previous work on ESP) has had a case of late-career integrity and has decided to play a wonderful joke on the all-too-deserving field of psychology by doing a few experiments until obtaining statistically meaningful results that defy everything we know about the universe. As I’ve pointed out before, one of the many problems with hypothesis testing is that you don’t have to try all that hard to prove anything, even when done “correctly,” given that nobody really keeps track of negative results.