Verizon and Netflix problems resolved by routing around Cogent?

By now the problems between Verizon and Cogent are well known, at least to nerds. FiOS users, especially on the East Coast, have been complaining of increasingly poor Netflix streaming performance. The reason stems from the saturated connections between Cogent and Verizon which neither refuses to fix; see here for a good background on the dispute. Basically, Cogent is sending more data to Verizon than vice versa, and Verizon is asking Cogent to pay for the upgrades required. The reason this story has been getting any press is that it highlights the complete insanity of the commercial internet system: ISPs expect to get paid to carry data packets, even if those data packets are requested by their own customers. In this case, Cogent is simply providing Verizon with the Netflix data that Verizon’s customers have asked for.

To highlight the ludicrousness of the way the internet operates, Verizon could presumably generate traffic from Cogent for which it expects Cogent to pay by issuing requests to download data from Netflix itself. Or, as Netflix  has pointed out, Netflix could resolve this situation by deciding to host its users backup data for them simply to artificially generate traffic going the other way. In fact, I’m surprised that Netflix doesn’t just program its streaming clients to repeat every bit back that they receive. That would solve this ludicrous problem, while also highlighting the stupidity of the way peering arrangements are made. At the bottom of this insanity is the fact that the companies who run networks have decided that they should get paid to carry packets like shipping companies would charge to carry packages. I would say it’s like UPS deciding to charge Amazon for shipping a package, while also deciding to charge the recipient for driving to their street. However, that’s not a perfect analogy, because if it were really like the Internet, UPS would be willing to waive the shipping if I handed them something to send back to Amazon. In fact, I struggle to find an analogy with the physical world of shipping, because there is no good analogy. Which is why it’s so incredibly stupid that network providers insist on billing arrangements that are analogous to shipping contracts.

Anyway, back to the point of this post: Comcast, which until recently had similar issues, has resolved them by getting Netflix to pay Comcast to connect directly to Netflix. There has been speculation Verizon would do the same. On the other hand, Verizon is probably not as willing to come to a reasonable solution as Comcast was, the latter trying to play nice to appease anti-trust regulators given it’s recent purchase of Time-Warner. I recently noticed an improvement in Netflix performance on FiOS, and wondered if maybe I was wrong about this. However, running a traceroute makes it clear that what happened is a third option I hadn’t considered; traffic between me and Netflix is going around Cogent and all the way to California:

3 g0-10-2-5.bstnma-lcr-21.verizon-gni.net (130.81.104.50)
4 ae1-0.bos-bb-rtr1.verizon-gni.net (130.81.151.60)
5 0.ae11.xl3.nyc1.alter.net (152.63.20.69)
6 0.xe-2-1-6.xt1.dca5.alter.net (152.63.0.113)
7 0.xe-4-1-3.xl3.iad8.alter.net (152.63.3.142)
8 tengige0-6-4-0.gw1.iad8.alter.net (152.63.35.145)
9 teliasonera-gw.customer.alter.net (152.179.50.234)
10 ash-bb4-link.telia.net (80.91.252.98)
11 las-bb1-link.telia.net (80.91.246.71)
12 netflix-ic-300871-las-bb1.c.telia.net (213.248.95.34)
13 ipv4_1.lagg0.c048.lax004.ix.nflxvideo.net (198.38.96.157)

Is it possible that rather than deal with Cogent or Netflix, Verizon has decided to just send East Coast Netflix traffic all the way to servers based in California, and to get there without using Cogent? Or, could Netflix have caused this by having clients make requests to different servers to get around the limited Cogent-Verizon ports? One thing that is clear is that we need a fundamentally different model for commercial internet if games like this are being played.

Stephen Wolfram is killing Mathematica

One of the saddest consequences of Steven Wolfram’s descent into megalomaniacal insanity (vis his decision to save science from itself by reinventing it in the image of a popular science book from the 1980s) is the continuing decline of Mathematica, his greatest (and, he seems intent upon forgetting, only) accomplishment.

Why the return to bitter posts? The week of my life I’ll never get back trying to get Mathematica‘s pitifully bad graph theory functions to yield correct results. I never thought I’d see the day when I considered MATLAB a superior product to Mathematica for doing something like network theory, but that day has come. I could go into great detail on the poor design of Mathematica‘s Graph object, but I’ll just leave the reader with the following object lesson on the perils of letting one’s ego interfere with one’s day job:

This is what happens when you decide to reinvent science but instead rediscover incompetence.

This is what happens when you decide to reinvent science but instead rediscover incompetence. (Note: the second graph has a lower “shortest path” despite losing an edge.)

Another nice bug is the fact that WeightedAdjacencyGraph[WeightedAdjacencyMatrix[g]] often returns an error, despite the obvious fact that it should return the original graph (at least topologically).

Seriously, Wolfram. Are there many more important mathematical topics today than graph theory? You can’t throw a copy of Mathematica these days (and I plan to) without hitting somebody working on a topic for which graph theory plays a central role. The fact that the interface to Graph[] is an embarassment is nothing compared to the fact that it doesn’t even return correct results when things like GraphDistance[] are applied to a graph which has been manipulated. When Mathematica starts returning mathematically incorrect results, something is wrong with the world. That thing, I believe, is Stephen Wolfram himself. It’s time for him to move on from Wolfram and let somebody else run the show.

For the few still reading and interested: the bug shown in the picture appears randomly. Most of the time you run that code it yields a plausible, maybe even correct, answer. The oddness of the “road network” graph shown is intentional. The application I was working on was an algorithm to generate a statistical family of road networks. My plan was to create a nearly fully connected (not even planar) graph and then prune the edges which were economically infeasible (didn’t yield a change in the shortest path sufficient justifying their existence relative to some parameter). It was in the process of trying to implement the pruning algorithm that I discovered this bug.)

I had originally made plans to move this to a private server, but since I’m working at MIT Lincoln Lab, it looks like I’m going to be able to keep all of my MIT computer access from when I was a graduate student. So, this will stay up indefinitely. I know you’re relieved. I may even start writing posts again…

How to rotate iPhone video on a Mac

I ran into what is apparently a not uncommon problem with iPhone video: you start to take a video while the camera is in portrait mode (just by accident of how you’re holding it) and then the rest of the video is stuck that way, even if you took 99% of it in landscape mode. If that didn’t make any sense, the bottom line is I had a video I needed to rotate 90 degrees. While there were plenty of solutions available on the PC, tons of Googling turned up virtually nothing for Mac Os X, short of finding an old copy of iMovie from five years ago.

Fortunately, I lucked in to a great solution, which doesn’t even require transcoding (with the attendant loss in quality that would result). This worked for me in getting a video from portrait to landscape, and I suspect it will only work in that situation. (This should be the only situation in which this problem occurs, as nobody in their right mind should ever shoot video in portrait mode on purpose, and if you do, I’m certainly not going to be complicit in aiding and abetting that crime against humanity.)

The solution requires a copy of the very nice all-purpose video player, VLC.

  1. Open the video in VLC
  2. It should actually open up in landscape orientation, regardless of the erronious orientation data in the movie file from the iPhone.
  3. Select “Streaming/Exporting Wizard” from the File menu.
  4. Select “Transcode/Save to file” and click next.
  5. Use “Existing Playlist” and select the file you just opened below, click next.
  6. Leave everything untouched (i.e. both check boxes blank) on the transcode screen and click next.
  7. Choose MPEG4, click next.
  8. Click “Choose…” to tell VLC where to put the output file, click next.
  9. Click finish.

This should be all it takes. The process will be fairly quick, since there’s no transcoding, but its not instantaneous as it does have to move a lot of bits into a new file.

Proof you’re a parent

For some reason I decided to take a look at the top 50 most played songs on my iPod. Here they are, listed from 5 to 1.

Rank Title               Artist             Number of Times Played
5.   Many the Miles      Sara Bareilles     66
4.   Fireflies           Owl City           67
3.   The Wider Sun       Jon Hopkins        81
2.   Baby Monkey         Parry Gripp        322
1.   Oatmeal             Parry Gripp        393

One guess as to which two songs are Alex’s favorites?

The plan is coming together nicely. Almost.

Ok, I know I promised no more useless political banter, but I think I’ll grandfather in follow-up posts to old articles. A while back I wrote about my “conspiracy” theory that Obama pushed through a health care reform bill that he knew would be struck down as unconstitutional, to set himself up for a rewrite of the reform to be a UK-style single-payer system (which would be constitutional, since there’s nothing wrong with the government forcing you to buy stuff from them).

So far, the first part of my prediction has come true, at least in the sense that the bill was partly struck down. Of course, we’ll never know what Obama was really thinking, but I maintain that by far the most likely scenario is that he knew full well it would never pass muster. The man is extremely smart, especially for a politician, and he was a law professor. I’m certain he knows that having the Federal Government mandate that private parties enter into a contract is grossly unconstitutional (not to mention the mother of all slippery slopes). I could buy that Pelosi didn’t see this happening (she looks constantly surprised to be wherever she happens to be) but I can’t believe Obama didn’t.

I think if there is one flaw in their plan, it’s that Obama didn’t see the midterm elections going as badly as they did. That’s too bad. As I argued in my original post, single-payer is really the only way to handle health insurance that makes any sense. Forcing private insurance companies to take people they know are sick is insane. Forcing people to buy something from somebody as a condition of their very existence is patently unconstitutional, and always should be. (To head off the people who point out we are required to buy car insurance, that is a condition of driving on public roads, not being alive and American, and no car insurance company is legally forced to provide insurance to somebody with 27 moving violations and two vehicular manslaughter convictions.)

The idea that people should have a million dollar PET scan machine at their disposal health as a fundamental human right is intellectually bankrupt, but that doesn’t change the fact that it may nonetheless be the Right Thing to do if we can afford it. We can, at least for now, and so unless we’re going to let people go without care needlessly, the only way to get everybody health care that makes any sense is single-payer. I challenge anybody to propose an alternative solution that provides all citizens with quality health care but doesn’t open a pandoras box of legislative legerdemain required to enforce access.

Hypothesis testing proves ESP is real

Or maybe it would be more accurate to say that ESP proves frequentist statistics isn’t real. In what has got to be one of the best object lessons in why hypothesis testing (the same statistical method usually used by the medical research industry to produce the Scare of the Week) is prone to generate false results, a well-respected psychology journal is set to publish a paper describing a statistically significant finding that ESP works.

The real news here, of course, is not that ESP has been proven real, but that using statistics to try to understand the world is a breeding ground for junk science. If I had to guess, I’d say the author (who is a well-respected academic who has never published any previous work on ESP) has had a case of late-career integrity and has decided to play a wonderful joke on the all-too-deserving field of psychology by doing a few experiments until obtaining statistically meaningful results that defy everything we know about the universe. As I’ve pointed out before, one of the many problems with hypothesis testing is that you don’t have to try all that hard to prove anything, even when done “correctly,” given that nobody really keeps track of negative results.