Category Archives: Technology

Free AppleScript for finding duplicates in the Apple Photos App on OS X

If you have a huge Photos library on the Mac, you very likely have a bunch of duplicate photos. People have also found that duplicates photos are sometimes created when you upgrade from iPhoto or Aperture to Photos. Unfortunately, if you look at the “free” programs on the App Store, you find that they are generally either scams or teasers for a paid version.

So, I wrote an AppleScript to go through every photo selected in Photos (you can select your entire library easily, if you want) and create a new Album with just the duplicate photos found. The script can be copied to your Photos script folder (if you have the script menu enabled) or you can just run it from with ScriptEditor. I also created a small script to reveal the ID of a selected photo, which can be useful for debugging the library and/or finding an original file within the inscrutable Photos database. Most people (if anybody) will just want the Find Duplicates script. Download them at the link below:

Photos AppleScripts

Once the script finds any duplicates, it creates a folder called “Duplicates” and adds a new album with the found duplicates. You should then go through that album and confirm that the duplicates are legitimate. If they are, select which the one duplicate you want to remove (you won’t want to get rid of the entire pair, presumably), right click and select delete. Make sure you don’t delete both in the pair, but if you do it’s not the end of the world; they will stay in the deleted photos album for a month.

Let me know if something doesn’t work right. The Photos AppleScript library doesn’t allow any destructive operations other than deleting albums (which I don’t do in either script) so the worst case should be that if it doesn’t work it just doesn’t find duplicates that it should.

Update: As a couple folks pointed out in the comments, the next step will be to create a script that allows you to “merge” a pair of photos, moving all the album associations and tags onto just one. I haven’t looked into whether this can even be accomplished with the Apple Script hooks available in Photos, so if anybody has an insight I’d appreciate it.

Verizon and Netflix problems resolved by routing around Cogent?

By now the problems between Verizon and Cogent are well known, at least to nerds. FiOS users, especially on the East Coast, have been complaining of increasingly poor Netflix streaming performance. The reason stems from the saturated connections between Cogent and Verizon which neither refuses to fix; see here for a good background on the dispute. Basically, Cogent is sending more data to Verizon than vice versa, and Verizon is asking Cogent to pay for the upgrades required. The reason this story has been getting any press is that it highlights the complete insanity of the commercial internet system: ISPs expect to get paid to carry data packets, even if those data packets are requested by their own customers. In this case, Cogent is simply providing Verizon with the Netflix data that Verizon’s customers have asked for.

To highlight the ludicrousness of the way the internet operates, Verizon could presumably generate traffic from Cogent for which it expects Cogent to pay by issuing requests to download data from Netflix itself. Or, as Netflix has pointed out, Netflix could resolve this situation by deciding to host its users backup data for them simply to artificially generate traffic going the other way. In fact, I’m surprised that Netflix doesn’t just program its streaming clients to repeat every bit back that they receive. That would solve this ludicrous problem, while also highlighting the stupidity of the way peering arrangements are made. At the bottom of this insanity is the fact that the companies who run networks have decided that they should get paid to carry packets like shipping companies would charge to carry packages. I would say it’s like UPS deciding to charge Amazon for shipping a package, while also deciding to charge the recipient for driving to their street. However, that’s not a perfect analogy, because if it were really like the Internet, UPS would be willing to waive the shipping if I handed them something to send back to Amazon. In fact, I struggle to find an analogy with the physical world of shipping, because there is no good analogy. Which is why it’s so incredibly stupid that network providers insist on billing arrangements that are analogous to shipping contracts.

Anyway, back to the point of this post: Comcast, which until recently had similar issues, has resolved them by getting Netflix to pay Comcast to connect directly to Netflix. There has been speculation Verizon would do the same. On the other hand, Verizon is probably not as willing to come to a reasonable solution as Comcast was, the latter trying to play nice to appease anti-trust regulators given it’s recent purchase of Time-Warner. I recently noticed an improvement in Netflix performance on FiOS, and wondered if maybe I was wrong about this. However, running a traceroute makes it clear that what happened is a third option I hadn’t considered; traffic between me and Netflix is going around Cogent and all the way to California:

3 g0-10-2-5.bstnma-lcr-21.verizon-gni.net (130.81.104.50)
4 ae1-0.bos-bb-rtr1.verizon-gni.net (130.81.151.60)
5 0.ae11.xl3.nyc1.alter.net (152.63.20.69)
6 0.xe-2-1-6.xt1.dca5.alter.net (152.63.0.113)
7 0.xe-4-1-3.xl3.iad8.alter.net (152.63.3.142)
8 tengige0-6-4-0.gw1.iad8.alter.net (152.63.35.145)
9 teliasonera-gw.customer.alter.net (152.179.50.234)
10 ash-bb4-link.telia.net (80.91.252.98)
11 las-bb1-link.telia.net (80.91.246.71)
12 netflix-ic-300871-las-bb1.c.telia.net (213.248.95.34)
13 ipv4_1.lagg0.c048.lax004.ix.nflxvideo.net (198.38.96.157)

Is it possible that rather than deal with Cogent or Netflix, Verizon has decided to just send East Coast Netflix traffic all the way to servers based in California, and to get there without using Cogent? Or, could Netflix have caused this by having clients make requests to different servers to get around the limited Cogent-Verizon ports? One thing that is clear is that we need a fundamentally different model for commercial internet if games like this are being played.

Stephen Wolfram is killing Mathematica

One of the saddest consequences of Steven Wolfram’s descent into megalomaniacal insanity (vis his decision to save science from itself by reinventing it in the image of a popular science book from the 1980s) is the continuing decline of Mathematica, his greatest (and, he seems intent upon forgetting, only) accomplishment.

Why the return to bitter posts? The week of my life I’ll never get back trying to get Mathematica‘s pitifully bad graph theory functions to yield correct results. I never thought I’d see the day when I considered MATLAB a superior product to Mathematica for doing something like network theory, but that day has come. I could go into great detail on the poor design of Mathematica‘s Graph object, but I’ll just leave the reader with the following object lesson on the perils of letting one’s ego interfere with one’s day job:

This is what happens when you decide to reinvent science but instead rediscover incompetence.

This is what happens when you decide to reinvent science but instead rediscover incompetence. (Note: the second graph has a lower “shortest path” despite losing an edge.)

Another nice bug is the fact that WeightedAdjacencyGraph[WeightedAdjacencyMatrix[g]] often returns an error, despite the obvious fact that it should return the original graph (at least topologically).

Seriously, Wolfram. Are there many more important mathematical topics today than graph theory? You can’t throw a copy of Mathematica these days (and I plan to) without hitting somebody working on a topic for which graph theory plays a central role. The fact that the interface to Graph[] is an unholy mess is nothing compared to the fact that it doesn’t even return correct results when things like GraphDistance[] are applied to a graph which has been manipulated. When Mathematica starts returning mathematically incorrect results, something is wrong with the world. That thing, I believe, is Stephen Wolfram himself. It’s time for him to move on from Wolfram and let somebody else run the show.

Looks like we’ll be here awhile…

I had originally made plans to move this to a private server, but since I’m working at MIT Lincoln Lab, it looks like I’m going to be able to keep all of my MIT computer access from when I was a graduate student. So, this will stay up indefinitely. I know you’re relieved. I may even start writing posts again…

How to rotate iPhone video on a Mac

I ran into what is apparently a not uncommon problem with iPhone video: you start to take a video while phone still thinks it’s in portrait orientation (long axis of the phone vertical) and then the rest of the video is stuck that way, even if you took 99% of it in landscape mode (the way God intended). You thus have a video that the Mac always wants to display sideways. If that didn’t make any sense, the bottom line is I had a video I needed to rotate 90 degrees, and maybe you do, too. While there were plenty of solutions available on the PC, tons of Googling turned up virtually nothing for the Mac, short of finding an old copy of iMovie from five years ago.

Fortunately, I lucked in to a great solution, which doesn’t even require transcoding (with the attendant loss in quality that would result). By the way, this worked for me in getting a video from portrait to landscape, and I suspect it may only work in that situation. (That said, this should be the only valid situation in which this problem occurs, as nobody in their right mind should ever shoot a video in portrait mode, and if you do, I’m certainly not going to be complicit in aiding and abetting that crime against humanity.)

The solution requires a copy of the very nice all-purpose video player, VLC.

  1. Open the video in VLC
  2. It should actually open up in landscape orientation, regardless of the erronious orientation data in the movie file from the iPhone.
  3. Select “Streaming/Exporting Wizard” from the File menu.
  4. Select “Transcode/Save to file” and click next.
  5. Use “Existing Playlist” and select the file you just opened below, click next.
  6. Leave everything untouched (i.e. both check boxes blank) on the transcode screen and click next.
  7. Choose MPEG4, click next.
  8. Click the button to tell VLC where to put the output file, then click next.
  9. Click finish.

This should be all it takes. The process will be fairly quick, since there’s no transcoding, but its not instantaneous as it does have to move a lot of bits into a new file.

Finally, a good Subversion client for Mac OS X

If you don’t have a Mac, or don’t know what SVN is, please accept my apologies for this very directed post. To the one guy remaining, rejoice:

For the longest time, there has been no good SVN interface available on the Mac. Windows folks had TortoiseSVN, and Linux folks wouldn’t be caught dead using anything other than command line tools (or, git, for that matter). So, everybody was happy but us Mac folks.

A program called “Versions” has been available for a while, but it, sadly, epitomizes the style over substance sin that is so prevalent on the Mac. It’s got a beautiful interface, but it’s an interface to very little. Namely, it doesn’t support merging or branching, which is pretty much the most important reason for using a versioning system like SVN. If you’re not branching and merging, you might as well just use a good backup system, because that’s pretty much all you’re using SVN for at that point.

So, I was very excited to find “Cornerstone,” which was recently upgraded to support the slickest SVN interface I’ve seen on any platform. It’s as pretty as “Versions” and as powerful (if not moreso) than TortoiseSVN. It’s merge facility is the best approach I’ve seen, for example. It’s intuitive, and as you adjust the settings it automatically performs a trial merge and gives you the results in real time. Awesome.

They have a two-week trial, which is more than enough to get a feel for the product, it’s so simple and well-executed.

(By the way, they aren’t giving me anything for this. I wish they were, but I don’t have that kind of juice.)

Getting the most data speed out of your cell phone

You may have noticed there have been very few posts here. There’s a reason for that. The first and foremost is that sending my rants in to the void has not been as personally cathartic as I’d hoped. My other goal for the blog, which actually has been somewhat successful, was to simply provide a vehicle for putting information out on the web that I thought might be useful for people, and that I couldn’t find elsewhere. Based on the traffic stats, those posts have actually been worthwhile, and my only reason for not doing more of this kind of post has been that I’ve been too busy playing with my son, finishing up my projects at MIT, and trying to get a job (in that order).

So, going forward, I’m just going to focus on the second category of posts (though I reserve the right to devolve to the first occasionally). This blog was getting too negative, anyway. In that spirit, here’s a particularly useful trick I just figured out while sitting in a coffee shop working remotely.

I recently gave up my nice window office since I was feeling guilty about taking up a nice spot but only working part time. So, I’ve been doing a lot of work remotely, usually from a coffee shop given that working at home just isn’t very productive when there’s an adorable toddler running around begging to be hugged. So, I splurged and decided to start paying the extra $20 a month to use my phone as an internet connection for my computer. This is becoming a pretty common thing, and Sprint even offers phones that will create a WiFi network on the fly (I use Bluetooth with my iPhone). I expect this will become even more common once the iPhone hits Verizon, as Apple will reportedly allow this version of their phone to create WiFi hotspots, too.

I would typically just leave my phone laying flat on the table next to my laptop. However, giving it a minute of thought, this is actually pretty dumb, for two reasons. First, having the phone so close to the laptop is probably not smart, as computers are notorious spewers of electromagnetic interference at pretty much every frequency imaginable. In theory, they should be shielded, but nothing is perfect and between the memory data rates and the processor clock speeds, a computer pretty much has the cell phone spectrum covered directly, if not with overtones. So, keep the cell phone away form the computer at least a foot or so.

Most importantly, however, leaving the cell phone flat on a table is a bad idea because it puts the antenna horizontal, whereas cell phone signals are polarized vertically. (What this means, if you’re not a fan of electromagnetics, is that the electrons in the cell phone tower antenna are being shaken up and down, not side-to-side. Radio waves are really just a way of keeping track of how electrons interact with each other. Without anything interfering, the electrons in your cell phone’s antenna will be wiggled in the same orientation and frequency as those in the cell tower antenna. However, antennas are designed for their electrons to be wiggled in a certain direction (it’s almost always along the long axis of the antenna) and a cell phone’s antenna is oriented with the assumption that the user is holding it upright against their ear.) Once I realized this, I put my phone up against a nearby wall so that it was standing straight up and down (as if somebody were holding it) and my data rates nearly doubled.

So, if you’re using your cell phone as an internet connection, keep it a bit away from the computer and prop it up so it’s vertical. Keeping it vertical in your pocket probably isn’t a great idea, since your body is pretty good at blocking radio. If you find this helps, please let me know in the comments. Right now my experience alone isn’t very statistically significant, to say the least.