Author Archives: Jonathan

About Jonathan

Asst. Group Leader in aerospace at MIT Lincoln Laboratory.

The results of my 25th annual desktop Linux survey are in: it sucks implausibly and should be mercy killed at this point

Since at least 1997, I’ve annually installed desktop Linux (recently Ubuntu) either on a VM or a partition to see if it’s finally ready for serious use. Some years are better than others, but the answer is always “no”. This year is no different. I tried Ubuntu 23.10. and here are my experiences on the FIRST DAY:

  • Color scheme not syncing uniformly across apps
  • Apps taking nearly a minute to appear after clicking on them
  • Ubuntu insists on showing my encrypted volume as an option on the dock
  • MATLAB install won’t respond to mouse clicks and is unreadably tiny
  • WiFi randomly disappearing and only coming back after restart
  • Several different ways presented as options to install a major software package: flatpak, apt, snap, direct download. This is an impressive number of options for a desktop OS with nearly zero market share. It takes some real zealotry to take something with no market share and divide it further!
  • Inability to update 1Password using snap.
  • Networking broken under Docker due to complex networking configuration using local cache on loopback interface. Fixing required manual editing of firewall to allow Docker’s backup workaround to work.

What’s especially disappointing is that this year Ubuntu feels slower than Windows. One of the things that has always attracted me to linux is the resource efficiency and all-around “quickness” of it. At this point, however, Windows actually feels faster and more responsive than linux, at least Ubuntu’s version.

How can desktop Linux be going this long and still suck so goddam badly? How can it remain such a complex mess of brittle configuration files and ever changing standards? How can it be so fractionated across so many distributions and software stores when nobody uses it?

If desktop Linux were a dog (and figuratively it is) it would be put down long ago. I love unix so I so want linux to be good, but it just isn’t. On the server it’s amazing. On the desktop it’s embarassing. And it makes me angry, because before linux destroyed the desktop unix market without offering a viable replacement, we had options. HPUX was good. Solaris was good. They all worked reasonably well. They were built by serious people. The people that build desktop linux can only be described as indecisive but talented engineers with vastly undermedicated ADHD and perhaps indiagnosed borderline personality disorder.

Like the definition of insanity, however, they keep doing the same things year after year and expecting a different outcome. Market share for desktop linux continues to bounce along the bottom and yet they keep introducing more distributions, fighting over software packaging formats like academics squabbling over something, well, academic. And like academics, the competition is so cutthroat perhaps because the stakes are so low. But really, who cares about which distribution wins a war nobody is bothering to show up for anymore?

Let’s just call time of death, ok? Linux is a server OS, not a desktop one. I don’t know why it’s possible for one to be effective as an open source project and the other to be an unmitigated disaster, but I suspect it partly has to do with the fact that taking unix out of the server world and into end-user computing was always a stretch even in the best scenarios. The only way to make a bad idea work is usually lots of talent and money, and you don’t get either for free. Whatever the reason, I’m officially done. And I mean it this time. This is the last time I uninstall a desktop linux distro in disgust, and the last time I install one in naive hope.

MIT’s Kornbluth sets a new low for academic “leadership”

I’m embarrassed to be an MIT alum and employee. Our president, in a recent testimony in front of congress, couldn’t manage to answer the question “is it ok to call for genocide on your campus?” without consulting attorneys and her colleagues at other institutions.

I’m not going to jump on the bandwagon of people calling for her to resign over this, nor will I criticize her late discovery of free speech. To be fair, while I find it disturbing that the “words are violence” crowd thinks threatening actual violence is speech when it’s against Jews, I can understand people supporting the idea that we should err on the side of tolerating free speech as much as possible, even if I don’t think it qualifies as such in this case. And certainly what’s happening in Gaza is unconscionable beyond any question of whether it’s justified.

That said, regardless of how you feel about Kornbluth’s answer, what she displayed was a lot of things but none of it was even remotely leadership. And that’s unequivocal.

When somebody asks you, as the leader of a university, a question about your academic policy, let alone a softball question like “you cool with calls for intifada against Jews?” you should be able to speak for yourself. You shouldn’t have to look over at Harvard to copy their homework, or consult with a team of lawyers about what you should say.

I suspect college and university would be a lot more affordable for Americans if we just outsourced college administration to a centralized management service. (Perhaps the DNC?) All college administrators think the same at this point and copy each other’s actions and policies, so why do we need to pay for more than one of each?

If we’re going to have zero intellectual diversity in academia, at least we could gain some economy of scale from it and get some relief for families struggling to pay for tuition at the same time universities increasingly struggle to justify their value proposition.

Is it safe to buy refurbished hardware?

I recently bought a refurbished router, the same model I’ve used elsewhere in my house with no problems. With THIS one, however, I found myself unable to allow SSH traffic to pass the switch, even with the router in bridge mode. It’s very weird, and probably nothing, but it made me wonder: wouldn’t a great cyber implant strategy be to buy router equipment, replace one or more of the chips with a compromised version that does some nefarious thing (e.g. opens ports to the outside, acts as a node in a DDoS attack, etc.) and then return it to the store for a refund? The store then takes it back, cleans it up and resets it (but doesn’t, of course, notice the new chip soldered in as they don’t have the process to check for that) and resells it as refurbished. If I were a state actor looking to get stuff behind people’s firewalls, refurbished items would be a great attack vector.

Given that pretty much every company that sells routers sources their chips from Asia, wouldn’t it be easier for a state actor to just put the implants in the chips themselves? Well, I don’t think so. That would be too risky and traceable, and eventually production spot checks would likely find something. The process for refurbished products, however, is much less rigorous, and anything found would be impossible to trace back since the product was out of the supply chain for so long.

Are women more honest than men?

The journal Science recently published a fascinating article from Alain Cohn et al, which looked at cultural proclivities for civic honesty around the globe. They employed a rather ingenious method: they “lost” wallets all over the world and recorded when the receiver of the lost wallet attempted to return the wallet to its rightful owner. The wallets were fake and included a false ID of a person who appeared to be local to the country in which the wallet was lost, including fake contact info that actually belonged to the researchers. The ingenious element of the research was that instead of leaving the wallet out in the open, the research assistants actually pretended to have found the wallets in our nearby local businesses and turned in the wallet to somebody working in that business, thus enabling them to record interesting ancillary data on the “subject,” such as their age, if they had a computer on their desk, and whether or not the person was local to the country. Clearly, the researchers were hoping to engage in a little bit of data mining to ensure their not insignificant efforts returned some publishable results regardless of the main outcome.

As it turns out, they needn’t have been concerned. The level of civic honesty, as measured by wallet return rates, varied significantly between cultures. In addition, there is an interesting effect where the likelihood of the wallet being returned increased if there was more money in it, an effect that persists across regions and which was evidently not predicted by most economists. I encourage you to read the original article, which is fascinating. On the top end of the civic honesty scale are the Scandinavian and Northern European countries, with rates at around 75%. On the bottom end of the curve is China, with about 14%. In the case of China, all the study did was confirm what anybody who does business there knows, and something that has been well covered by journalists and completely ignored by our politicians: to the Chinese, not cheating is a sign you’re not trying hard enough.

Here’s where things get interesting: in keeping with modern scientific publishing standards, the researchers made their entire dataset available in an online data repository so that others could reproduce their work. There are a lot of interesting conclusions one can make beyond what the authors were willing to point out in their paper, perhaps due to the political implications and the difficulty of doing a proper accounting for all the possible biases. However, unburdened by the constraints of an academic career in the social sciences, I was more than happy to dig into the data to see what it could turn up…

Perhaps the most interesting thing I found is that women appear to be more honest than men. Over the entire world-wide dataset, women returned the wallets about 51% of the time, versus 42% for men. It is tempting to look at individual countries, but the male versus female difference is not statistically significant enough when looking at individual countries, so I chose to only look at the aggregate data. The data is not weighted by country population, so one should take the absolute magnitude of the difference with a bit of skepticism. However, looking at the individual country data it appears a proper accounting for population bias would likely maintain or increase the difference. (Some of the most populous countries had the largest difference between women and men.)

Worldwide, women appear to be statistically significantly more honest than men. Standard error was less than 1% for both cases.

Here is the full dataset of men versus women broken down by country. You can see that the most populous countries are those where women appear to be more honest than men, so fixing the chart above to account for sample bias would likely still find a significant difference.

Women appear to be more honest than men in most cultures, though the individual country results are not usually statistically significant.

Another interesting question to ask of the data is whether or not there is a generational difference in honesty. Surprisingly, the answer turns out to be that there’s not a statistically significant difference:

Age doesn’t appear to be a statistically significant predictor of honesty. Standard error was roughly 1%, so the difference shown is not meaningful.

Looking at the breakdown by country, we see that there are no big differences between the generations, with one exception that I’m not even going to try to explain:

It’s possible that the young are more honest than the old, but it doesn’t appear to be statistically significant except in one country.

One interesting set of issues that always comes up with population studies like this is what, if anything, should we do with this information? It is true that a Swedish woman is about eight times more civically honest, on average, than a Chinese man. That’s interesting, but also pretty dangerous information. Should this inform our immigration policy, where population statistics might actually be valid? Is it better to not even ask these questions given the abuse of the information that might result? Or, is it good to have this information, especially when it flies in the face of our image of ourselves and others? I suspect in the case of the US, most would be surprised to find out that the average US citizen is as honest as the average Russian. We may be surprised by both halves of that statement, and both might be good to think about.

Is technical analysis and algorithmic trading a legal form of market collusion?

Despite all dictates of logic against it, technical analysis actually works some of the time. I’m not talking about the random luck of any scheme working, but the fact that an uncanny amount of the time, “support” and “resistance” theories seem to be borne out in actual stock data. Assuming I’m not just imagining this, the obvious answer as to why is that it’s a self-fulfilling prophesy. If enough people believe in technical analysis, the patterns inherent to its doctrine will occur because the market is just composed of humans and their beliefs. It’s no more odd than the fact that stocks go up when earnings go up, except that belief at least has some basis in fact. The point is, having a basis in fact is irrelevant to the market.

Also obvious, however, is that when a stock pattern really works, it will draw attention to itself and it will stop working as people try to capitalize on it. Thus, while it may be a self-fulling prophesy that technical analysis will work, the same logic suggests that it should also fail, on average, because the market is mostly a zero-sum game. So, what gives?

It occurred to me that we are missing one vital factor: there are a lot of people who think technical analysis is utter hogwash. Ironically, however, if enough people think technical analysis is bogus, it just might work. It then dawned on me what technical analysis may really be and why it might actually work: a legal form of market collusion.

Technical analysis may work out to be a somewhat effective form of conspiracy among a subset of the market participants. The rules of TA are essentially a language, allowing adherents of the art to manipulate the stock market through mass distributed collusion. I think it is entirely possible that the “rules” of technical analysis are successful because they manipulate the market in a way that gives an advantage to adherents of those rules relative to those who don’t.

The net effect is that a group of people in the market cause prices to oscillate by all “colluding” to buy and sell at certain points. Another group of people then end up making trades at what the other half know to be the “wrong times” in a predictable cycle. The common rules of TA form a system of collusion that doesn’t require any back channel conversation between participants, and thus it is completely legal; the market itself is the channel through which the mass collusion occurs. The general disregard among academics for the possibility that TA could work results in a situation where there are always enough market participants who aren’t in on the game so that the game works. In fact, perhaps the best thing that ever happened for TA was the publication of “A Random Walk Down Wall Street.”

Free program for finding duplicate photos in the Apple Photos App on OS X

UPDATE: This script is no longer necessary given Apple’s introduction of automatic duplicate detection to iOS. I’ll leave this post here for archival purposes, however.

If you have a huge Photos library on the Mac, you very likely have a bunch of duplicate photos. People have also found that duplicates photos are sometimes created when you upgrade from iPhoto or Aperture to Photos. Unfortunately, if you look at the “free” programs on the App Store, you find that they are generally either scams or teasers for a paid version.

So, I wrote an AppleScript to go through every photo selected in Photos and create a new Album with just the duplicate photos found. (With a large album I recommend selecting one or two years at a time to avoid issues with memories.) The script can be copied to your Photos script folder (if you have the script menu enabled) or you can just run it from with ScriptEditor. I also created a small script to reveal the ID of a selected photo, which can be useful for debugging the library and/or finding an original file within the inscrutable Photos database. Most people (if anybody) will just want the Find Duplicates script. Download them at the link below:

Photos AppleScripts

Once the script finds any duplicates, it creates a folder called “Duplicates” and adds a new album with the found duplicates. You should then go through that album and confirm that the duplicates are legitimate. If they are, select which the one duplicate you want to remove (you won’t want to get rid of the entire pair, presumably), right click and select delete. Make sure you don’t delete both in the pair, but if you do it’s not the end of the world; they will stay in the deleted photos album for a month.

Let me know if something doesn’t work right. The Photos AppleScript library doesn’t allow any destructive operations other than deleting albums (which I don’t do in either script) so the worst case should be that if it doesn’t work it just doesn’t find duplicates that it should.

As a couple folks pointed out in the comments, the next step will be to create a script that allows you to “merge” a pair of photos, moving all the album associations and tags onto just one. I haven’t looked into whether this can even be accomplished with the AppleScript hooks available in Photos, so if anybody has an insight I’d appreciate it.

Looks like we’ll be here awhile…

I had originally made plans to move this to a private server, but since I’m working at MIT Lincoln Lab, it looks like I’m going to be able to keep all of my MIT computer access from when I was a graduate student. So, this will stay up indefinitely. I know you’re relieved. I may even start writing posts again…