Based on what I’ve read from Aaronson’s blog, the arxiv preprint that describes what Willow is, dates from a few months ago.

The release is mostly a naming thing and the theoretical progress comes from the error correction from the link. I assume it’s hard to assess Willow’s practical value in daily life without offending the people building it while being intellectually honest to laymen.

What can be said from what I’ve seen is that the timeline of QC progress is steady and that Willow (like it’s predecessor, Sycamore) can run some operations (not as simple as like addition or substraction, closer to “is this gigantic file of random numbers produced by a natural coin flip or a fake one?”) that your gaming PC can’t in many years 😊

I’m by no means a QC expert, just a habitual reader of LessWrong-related blogs (and I blog myself to improve my knowledge and store it for later use) and my concept of a QC in the real world would look like a “boring” lab circa the year 2040 where a pharmaceutical company gets a contract to develop a patent for a pharmaceutical and uses a quantum computer. It’s not as sexy as Richard Feynman describes it.

The obsessive part of my brain noticed something new that’s related to quantum computers: the error correction zoo — which is beginning to get some maintenance work in this month!

It’s available on https://errorcorrectionzoo.org/ and editable on https://github.com/errorcorrectionzoo/eczoo_data/:

From what I’ve seen professors and researchers don’t remember things like specific details for an ECC in practice because there’s no incentive to (unless they are teaching/using them in that particular time). You can think of the online zoo as an informal encyclopedia of ways to correct errors in messages.

Analogous to how veterinarians take care of animals in normal zoos, the “vets” of the error correction zoo take care of all the error-correcting codes.

What makes the zoo interesting, and related to why quantum computers don’t work well in practice yet, is that error correction (at scale) is still an unsolved challenge for quantum computing. Creating a classical bit is easy: anything that can store true/false over time is good.

Creating a qubit, is, well harder, but not impossible — think of any quantum phenomenon from HS physics intro like light polarization and you can represent a quantum state — that’s a physical qubit. But, nature loves to fiddle with quantum states and break them via noise — so your physical qubit isn’t reliable. And we have to find ways to fix them.

The ECC zoo is maintained by Victor V. Albert from NIST, Philippe Faist from Caltech and many others. I don’t know the specifics of the error correction that Willow uses, but the core idea is the Kitaev surface code introduced in 2003. Once you have enough noisy qubits in the real world which we will call physical qubits, we you arrange them in a square grid and use some of them for redundancy Kitaev’s code and get logical qubits. In theory, f you have enough logical qubits (which we unfortunately don’t with Willow), you can run the Shor’s algorithm that appears in movies. But — one day.

You can read this excellent Quanta Magazine on Google’s Willow progress, as well as error correcting codes here: https://www.quantamagazine.org/quantum-computers-cross-critical-error-threshold-20241209/

P.S. The ECC Zoo can be cited as:

@book{ErrorCorrectionZoo,
  title={The Error Correction Zoo},
  year={2024},
  editor={Victor V. Albert and Philippe Faist},
  url={https://errorcorrectionzoo.org/}
}

It is inspired by the related “CS zoo” called Complexity Zoo started in 2002 that contains 500+ classes from complexity theory:

Then, 8 years ago, Professor Brailsford of Computerphile fame, explained the Busy Beaver game in video format (you can watch the video, and it involves the Busy Beaver game for some n: an abstract computer i.e. a Turing Machine is some abstract machine that can be in n different states, and the goal is to find a machine that works hard changing its memory without ending in an endless loop):

Throughout his video, he relates the beaver game to the function BB(n) also known as S(n): the function that counts the maximum steps for a game with n states and he goes through the first few “scores” or steps. At 13:31 he mentions that he doesn’t even wanna calculate how many Turing machines there are for n=5, much less calculate the value of the max steps.

Well, in 2022, Aaronson conjectured that BB(5) = 47,176,870 in a paper that analyzes the low-hanging fruits (as well as more difficult problems and challenges) regarding this fascinating function.

Then, some people (initiative by Tristan Stérin, [edit] but he messaged me to mention all the 20+ highly collaborative people that are producing the results!) started bbchallenge.org to solve it. You can read the story here — they read Aaronson’s paper and decided they will write independent deciders, i.e. programs that will decide the behavior of families of machines that are well-known.

This month we’ve got some proofs that are ready to be analyzed that in fact prove this.

As Tristan Stérin says:

Nonetheless, bbchallenge’s endgame approaches and we’re very excited for what comes next!

Beaverly yours,

If you want to help solve BB(5) (by taking a look at the proof or otherwise) or even take a look at BB(6), you can also join the Discord server.

On March 12 2024, Cognition Labs, a novel name for the public (but with the #1 gold IOI ‘14 medalist behind it) showed a demo of Devin - the “AI-powered software engineer”:

The only way to access Devin is to sign up on a form/waitlist, so we cannot be sure how well this video demonstrates the capabilities — but let’s analyze their work, the investors, the team and the claims on Twitter.

They claim they found a job on Upwork.com (related to computer vision and Python) and made Devin complete it:

… Devin fails due to a PyTorch version issue, fixes it, asks a few questions about the output and fully completes the task! This is impressive, but it doesn’t end there. One of the extra videos shows a complex case where the user is supplying Devin a link to a GitHub repo that’s an old fork of the math library SymPy with a really difficult bug in logarithm calculations:

Devin clones the repo, analyzes the whole SymPy codebase (!) and finds a division error related to the code for calculating logarithms, fixes it and even runs tests related to logarithms to make sure that the fix doesn’t affect other parts of the project:

And sure enough, if you look at SymPy this is the real world issue “#17148: Incorrect extraction of base powers in log class”.

Considering it’s an issue from 2019, is there a chance it’s in the training dataset? I don’t know, but honestly it’s not even important. What’s truly important is how Devin handles big contexts and how it works on such a huge repository. SymPy is a very complex project! My guess is that their magic sauce lies in iterative search and analysis of repositories when feeding the most important details to a transformer/LLM architecture.

Who’s behind this?

The supporters and investors include the Thiel’s Founders Fund, Doordash co-founder Tony Xu, Stripe CEO and cofounders Patrick Collison and John Collison, the founder of Modal Labs (the same Modal which I mentioned in a previous post), and many others.

As for the founders - Scott Wu is a Harvard graduate and Steven Hao is an MIT graduate, and they already have professional experience with software engineering and machine learning. They also have competitive programming experience - 10 International IOI (Olympiad in Informatics) gold medals, which I admire (timed tasks on difficult problems are not for everybody 😄).

People on social media quickly noticed that Wu was a legendary master on Codeforces and, as a small kid, used to be a really, really fast MATHCOUNTS champion — to the point where he acts like a math-expression autocomplete.

Do I think the demos are cherry-picked?

70% no, 30% yes. I’m sure that setting up or re-building many projects requires painful commands with package managers (and all the dependency hell experiences that engineers are used to), external and shared libraries, and obtaining tokens on third-party services that are outside of the scope of what state-of-the-art machine learning assistants can do.

That being said, if a human does the painful things - ML assistants will dominate at many tasks. Boilerplate code is definitely going to be a solved problem. A huge amount of debugging and compiler error tasks can be automated and some are even suited better for computers than humans - especially those revolving generics, floating point math, nested loops with many index variables and macros.

What is SWE-Bench?

A key selling point they claim is 13.86% unassisted score on SWE-Bench. So what’s SWE-Bench?

SWE-Bench is a paper and benchmark from Princeton University and the University of Chicago that contains instances of unsolved (GitHub) issues. The input is the whole codebase and a human-readable issue, and the expected output is a patch to the codebase. Fundamentally, the task is to solve the issues without breaking the tests. The benchmark is really hard and approaches using Claude and GPT-4 can solve 0-2% of the task:

… This is why SWE-Bench also has an assisted version where a human can intervene - which can get Claude and GPT-4 to 4%. But Devin claims 13.86% unassisted score. I find this impressive, but I also think it’s a benchmark that’s too recent, since it comes from ICLR 2024 and many projects will be in the 4-6% range once it becomes more mainstream and well-recognized.

Conclusion

I hope we get access to Devin soon so that we can evaluate it without guessing, but I’m optimistic about its performance. I think it’s way, way too early to claim that software engineering has been automated or obsoleted, and I think Devin and similar tools are just going to be tools that are going to accelerate talented engineers.

On top of that, if you are scared, consider the fact that software engineering is not just about making pull requests related to a small subset of types of bugs, it’s also about planning, negotiating, meeting clients’ needs and talking to non-engineers.

In any case, I wish Cognition Labs a successful launch!

Although Macedonia is sometimes known as the country that produced fake news during the 2016 US presidential election, it has a positive phenomenon regarding online news that fascinated me during my teenage years. The most popular local website1 is TIME.MK – a news aggregator that manages to aggregate credible online newspapers in a transparent and user-friendly fashion that I hadn’t seen before – especially since there’s no local Google/Yahoo News. And citizens love it, and many attempt to create clones of it.

TIME.MK was developed by a person named Igor Trajkovski. In this article, I’ll talk to him2, find out how it was launched and learn about his background, inspirations and values.

Filip: Before talking about your background and your projects, I’d like to ask: What is TIME.MK? If we were stuck in a 2-minute elevator conversation, how would you describe it?

Igor: TIME.mk is a news aggregation platform that collects news articles from the biggest news websites in North Macedonia and by using the techniques from Natural Language Processing and Computational Journalism computes the top stories of the day. The way it is done is by first finding the stories, in the collection of news, by using techniques of document clustering and then ranking them according to the general criteria and proprietary algorithms.

The general criteria are: Bigger clusters are preferred, newer news are preferred, news from authoritative sources are preferred and diversified stories/clusters (by the number of sources that cover it) are preferred. The algorithm that runs behind TIME.mk is an amalgam of these four principles, and many other heuristics, that were learned through the 15 years of running the news aggregator.

Filip: The concept of recent news-first and ordering by cluster size make sense. Without leaking the whole pipeline or magic sauce behind TIME.MK, can you share an example of an algorithmic technique or a class of techniques you've been using?

Igor:  For clustering, we employ a highly optimized and tailored version of the Hierarchical Agglomerative Clustering (HAC) algorithm, with specific stopping criteria adjustments for merging. To measure the distance between clusters, we use the Word Mover's Distance Algorithm, which leverages word embeddings trained on an extensive corpus of 20 million news articles from 2001. In their basic forms, both algorithms are quite slow, making it challenging to complete the entire cycle of crawling, clustering, and ranking of 10k news in under a minute. However, through intelligent caching and optimizing recurring tasks in each cycle, achieving this speed becomes feasible.

Filip: 15 years is a long time. What was the inspiration for building it, and how did it start? I know it had something to do with your personal decision to move outside Macedonia.

Igor: Yes. Living abroad from 2001 to 2008, I was interested in Macedonian affairs. I regularly browsed various news portals like A1, Dnevnik, Vecer, etc. I noticed that different sources highlighted different top stories and some even censored certain events or topics. To get a comprehensive view of major events, one had to visit all the major news outlets. Google News, which emerged post-9/11, addressed this issue but lacked a Macedonian edition, a common scenario even for larger countries at the time.

This gap sparked my initial inspiration. I was curious whether the news aggregation concept would be effective in a smaller country like Macedonia and if text processing algorithms could adapt to the Macedonian language. To be honest, I also looked for a solution to the problem of passive income, the real application of IT in general and AI in particular, while spending the abundant free time reading AI papers/news, philosophy, psychology and history.

Filip: What were the biggest challenges during these 15 years?

Igor: TIME.mk was launched in July 2008. Initially, during the first three years, it wasn't the most visited website. Its concept of news aggregation seemed to resonate primarily with a more educated audience. Many, even to this day, mistakenly perceive TIME.mk as a news source rather than a tool for accessing news. This misconception even led some news editors to believe we were plagiarizing their content, raising concerns about intellectual property infringement, though legal action was never pursued.

A pivotal moment in TIME.mk's history occurred in 2011, an event I liken to the mammalian rise 65 million years ago. This was when the major news source and largest TV station, a1.com.mk, was shut down. This led to the emergence of numerous opposition news websites, formed by former A1 journalists. However, none could match A1's comprehensive coverage. During Macedonia's particularly polarized political climate, TIME.mk stood out as the only platform offering real-time coverage of events from diverse political perspectives, attracting visitors across the political spectrum.

Since then, TIME.mk has been the most visited website in Macedonia. We've expanded our services, adding features like TopTweets analysis from the Macedonian Twitter sphere, aggregating TV shows and political debates available on YouTube, and developing efficient search engines for public procurements. These enhancements have further strengthened our position in the news market.

We encountered two significant technical challenges. The first was algorithmic: clustering similar news stories, even when sources use varied vocabulary for the same topics. Modern NLP tools, particularly word embeddings, were instrumental in overcoming this challenge. The second challenge related to infrastructure, specifically DDOS (Distributed Denial of Service) attacks.

Being a major news source, we attracted various attackers, both local and international, motivated by monetary or political gain, aiming to disrupt our service. Developing robust defenses against these attacks posed a considerable challenge.

Filip: You mention that you lived abroad from 2001 to 2008. Reading your CV, you interned at Google and did research at Max Planck Institute for Computer Science in Germany and Jožef Stefan Institute in Slovenia. I’d like to know a few things:

1. The spicy one… What’s your opinion on publish-or-perish, p-hacking, grant and tenure negotiations in academia?

2. Do you have a favorite professor, course or a story during these years?

3. How does academia compare to the work at Google, and are R&D labs (ex. Microsoft Research or what used to be Google Brain) a viable alternative too? I’ve heard that some people choose the academic path instead of engineering because they can’t stand the deadlines and meetings :)  

Igor: 

1. I despise it. That 'publish-or-perish' mentality was one of the major reasons why I left my professorship, along with other issues related to how FINKI was managed. FINKI is the largest state Faculty for Computer Science in Macedonia. I never imagined that a research job could feel like working on a factory line, where you're expected to produce a certain number of papers within a set timeframe. This inevitably leads to unethical practices like p-hacking, plagiarism, self-plagiarism or recycling, and citation manipulation, all of which erode personal integrity. I'm ambivalent about the tenure system. It might be beneficial in social sciences, where it can protect professors' freedom to critique society without fear of losing their jobs. However, it also can be exploited by less productive professors who merely enjoy the status without fulfilling their responsibilities. This was a noticeable pattern at FINKI.

2. My favorite professor during my undergraduate studies was Prof. Dr. Oliver Popov. He taught an introductory course in computer science, focusing on computability and the philosophy related to the field. It's challenging to articulate the profound impact he had on us, instilling lifelong motivation and presenting computation as a reality-altering force. My favorite courses were 'Algorithms, Part I and II' from Princeton University, which I believe are fundamental to everything a computer scientist or programmer will ever build.

3. I've never held a research position in the industry, so I can't make a direct comparison. However, the research I conducted for my PhD was enjoyable. Although it required producing peer-reviewed journal papers, the timeframe was reasonable, and there wasn't excessive pressure or too many meetings. My industry experiences, including roles as a developer, left me less enthusiastic about agile development due to its frequent meetings and reports. Working in two Berlin startups, I found the division of responsibilities strange. In such environments, you're often expected to wear multiple hats – developer, tester, owner, dev-ops, etc. It was overwhelming to handle tasks beyond my expertise, like solving algorithmic problems, and also be responsible for implementing, testing, monitoring in production, and updating software as the environment evolved. There was an excess of tedious, uninteresting work. 

Filip: So if I understand you correctly, It seems like being aware of your own competitive advantages was the key when it came to experimenting between academia, engineering or starting a business.

You mention philosophy and computability, so that’s what I’m aiming at for the final questions. 

What’s your favorite philosopher and philosophy-related book, and do you have any thoughts on determinism, free will and gene editing?

Igor:

I'm particularly drawn to 18th and 19th-century German philosophers from the Enlightenment era. These thinkers were at the forefront of constructing meaning without relying on the concept of God. This challenge resonates with me as we, as humans, keep wrestling with questions about our place in the universe and our origins. Our search for meaning likely stems from a desire for control and predictability, necessitating a story or framework to make sense of our existence.

Among these German philosophers, Schopenhauer and Hegel stand out to me. From more contemporary thinkers, I highly recommend 'The Denial of Death' by the anthropologist Ernest Becker. Becker's core argument in this book is that the primary psychological function of culture is to offer humans ways to deny the inevitability and fear of death. He suggests that a significant portion of human behavior is driven by an unconscious fear of death, with cultural institutions and beliefs providing meaning, order, and a sense of permanence to our lives.

When discussing determinism and free will, it's crucial to first define these concepts in the specific context. I lean towards determinism, and thus, the perspective that free will is an illusion. Our representational and prediction systems are not capable of foreseeing the future, leading to this misconception. If the universe operates deterministically (as Einstein suggested, 'God does not play dice'), then free will seems implausible since everything is determined by previous states of reality. Even in a non-deterministic universe, the origin of free will remains unclear to me.

Regarding gene editing, I view it as an inevitable next step in the evolution of medicine, building upon past advancements like antibiotics, vaccines, prosthetics, stents, bypass surgeries, and pacemakers.

Filip: And for the final one. There’s a $10M prize for building an AI that wins an International Math Olympiad and there’s tons of buzz regarding ChatGPT and LLMs.

Also there are claims that AI-powered robots will beat pro soccer players soon.

If you were to make a rough guess (you may use uncertainty and probabilities as percentages), how would you answer these:

• How soon will the AI IMO prize be claimed?

• When will AI-powered robots beat pro soccer players?

• How likely is it that LLM/transformer-based solutions will be the first ones to solve a major human problem (including, but not limited to finding a cancer cure or solving unsolved math problems), or even worse, cause one (unaligned AGI that wants to hurt humanity)?

Igor:

I predict that within 3-5 years, we'll see AI entities, not humans, winning gold medals at the International Mathematical Olympiad (IMO) and the International Olympiad in Informatics (IOI).

Regarding AI-powered robots in professional soccer, my forecast is that in less than a decade, a team of non-human players will be capable of defeating the world champions in soccer.

As for the potential of LLM/transformer-based solutions in solving or exacerbating major human issues, including but not limited to cancer or unsolved mathematical problems, there's significant potential. Current LLM systems, including finetuning and RLHF, could indeed aid in finding solutions. However, I have a gut feeling that they're primarily interpolating within the confines of their training datasets, lacking the necessary guidance for meaningful extrapolation beyond this data. Genuine creativity requires the ability to extrapolate and explore, along with a set of values, whether explicit (like survival instincts) or implicit (such as ethical or moral values), to drive progress. Present-day LLMs lack both of these capabilities.

I’d like to thank Igor for his time replying to my questions and hope that TIME.MK will continue with its good work.

A startup recently asked me to help them design an infrastructure where they can:

• run pre-trained PyTorch model on some inputs, such that the execution happens after some customer actions, but shouldn’t last more than an hour

• try to avoid managing their own long-running GPU instances

• be able to re-deploy newer versions of the model and monitor if and when the model crashes

A lot of their infrastructure was hosted on AWS: S3 and DynamoDB for data, Cognito for authentication, but when it came to inference they were confused. A developer had built some prototypes with Lambda functions + SageMaker — the advertised solution for ML deployments, and it had some issues: Lambda execution time is limited at 15 minutes and configuring SageMaker’s environment became a messy pipeline which they couldn’t maintain1. A lot of the SageMaker models that worked had to run on a CPU. The SageMaker examples repository contains amazon-sagemaker-examples/tree/main/inference which is full of empty folders and ipynb notebooks with 404 links.

As a startup, they were fine with taking a risk of moving the ML inference to any platform2 (even platforms that are startups themselves) as long as they can get PyTorch running ASAP and the execution price is okay.

I already knew that people rent their GPUs on vast.ai at low rates, but even if the GPU owners signed their data processing contract, no one liked the idea of sending user data to hardware managed by strangers. I ran some quick Hacker News, Reddit, Google searches for cheap GPU inference and stumbled upon a comment mentioning Modal.com so I decided to check it out.

The pricing page was very clear and concise. As of the time of writing, it looks like this:

At first glance it's good, so I decided to compare them with AWS. A g4dn.xlarge would correspond to NVIDIA T4 and AWS would charge $0.000146/s while Modal is at $0.000164/s which is a 12% margin (same for hourly rates). That’s reasonable for us - as long as Modal helps us deploy and monitor our model without hassle.

By the way, if you haven’t run models in production and want to understand ML inference economics in practice, here’s an example: running OpenAI’s speech-to-text model (Whisper Large) given a 5 minute English audio input on an NVIDIA T4 would take around 30 seconds. That means a single Whisper call can cost $0.00438.

So I did the well-marketed pip install modal one-liner and logged in with a fresh account. It doesn’t require a credit card by default (yet?) and I got $30 free credit which could power our use for quite some time. Squaring an integer on a remote CPU is simple:

import modal

stub = modal.Stub("fiki-demo")

@stub.function()
def f(i):
    return i * i

@stub.local_entrypoint()
def main():
    # Call f locally
    print(f.local(1000))

    # Call f on Modal's cloud CPU
    print(f.remote(1000))

Just do modal run yourfile.py and it works. So how do I attach a GPU? Turns out it’s just a matter of editing the decorator:

- @stub.function()
+ @stub.function(gpu="T4")
def f(i):
    return i * i

And how do we ship dependencies like ffmpeg, our AWS secrets, install PyTorch and the pretrained model? It’s quite simple actually. Modal builds Docker images with your code attached, and you have full control over the Dockerfile - scroll to the bottom of the post for an example.

In addition, once you like how model run yourfile.py works, you can deploy your model to production and generate a REST API. The dashboard let’s you see some nice stats (as well as build logs, runtime logs, etc) and billing info.

A model with CUDA, torch, ffmpeg running on T4 cloud, assuming you have a local Python package with myapp/__init__.py and a do_something() model would look like this:

import modal
from modal import (
    Image,
    Mount,
    Secret,
    method,
)

image = (
    Image.from_registry(
        "pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel",
    )
    .pip_install(
        "openai==0.27.2"
    )
    .apt_install("ffmpeg")
)

stub = modal.Stub(
    "fiki-demo",
    image=image,
    secrets=[Secret.from_name("prod")],
    mounts=[
        Mount.from_local_python_packages("myapp")
    ],
)


@stub.cls(
    timeout=80000,
    gpu="T4"
)
class DiarModel:
    @method()
    def detect(self, data: bytes):
      import torch
      print("Is CUDA available:", torch.cuda.is_available())
      import myapp
      return myapp.do_something(data)

A call is just a matter of DiarModel().detect.remote(data) and deploying from CLI is just modal deploy yourfile.py.

If you can design functions that return JSON serializable data then you can also wrap them with @modal.web_endpoint(method="POST")and your deployment tab will show you the appropriate endpoint URL. Note: I haven’t checked the pricing on REST API bandwidth - if you have more information, please send me a message or comment: is it better to host big blobs on S3 or return them directly?

In conclusion, it’s been a pleasant surprise. At a 12% margin over raw AWS resources, you get a very nice development experience: a well designed CLI, dashboard, Google Colab-like responsiveness and a REST API (that boots GPU code within milliseconds) which you can use to get ML output.

It does seem like they are a young Series A company, which depending on your risk-aversion could mean they are not a suitable choice for you. Their status page indicates they utilize Google Cloud Platform, AWS and Oracle Cloud in order to power the code execution and they have >99.9% uptime so far:

I wish them luck and hope their platform stays developer-focused and reliable.

On August the 14th, Twitter user lisatomic posted the poll above. Just like the movie The Matrix, your job is to pick a red or blue pill — but this time it doesn’t look like the choice involves learning the truth about the universe. Or, that’s how it looks at first glance. It’s just a logic puzzle that involves survival.

The tweet containing the poll quickly gained thousands of retweets. Users familiar with mathematical puzzles noticed its similarity to the Prisoner’s dilemma. The poll, as well as the aforementioned dilemma are part of game theory, a branch of mathematics that studies strategic choices of two or more entities playing any game with well-defined rules.

If we want to find the optimal choice, we want to analyse the cost and benefit of each choice, while taking into account that other players are rational and will analyse the cost and benefit before making their choice too.

If we assume others are rational, they want to live — and the blue pill offers a world where you live as long as the majority picks it. On first hand, this sounds like a good idea, and that’s what over 64% of people did:

So we saved everyone, with a safety margin of 14.9%!

… But we are talking about life and death, and if you’ve read the Prisoner’s dilemma you know that strategic choices are nuanced, and in some cases the optimal personal choice is to betray everyone even though cooperation saves everyone.

In particular, users noticed the following fact: there is no case where a red pill dies.

• If the majority of users pick red, all reds survive.

• If the majority of users pick blue, everyone survives → all reds survive

It’s a fail-safe option. If everyone takes the fail-safe option, everyone is safe regardless of how manipulative and destructive others are.

At this point, people started arguing, with some blocking each other: how come it feels right to pick blue if it’s so wrong? Some lost faith in humanity and physicist David Deutsch remarked that it’s a gotcha question that involves skill that isn’t taught very often:

But there’s another point that needs to be made: game theory doesn’t involve mis-clicks, misreading the question, or the guilt of killing your grandma that chose the supposedly non-optimal blue. These social effects make a lot of sense in the altruistic camp:

The Canadian singer Grimes and the game designer Jonathan Blow think the final results 64.9% - 35.1% might not reflect reality:

In conclusion, this feels like the main character in Nolan’s Oppenheimer: it’s hard to articulate a coherent view when there are feelings involved. If you want to follow the latest discourse regarding this puzzle, you’d be interested in Roko Mijic’s Twitter timeline. If you like content like this, subscribe to the blog and follow me on Twitter.

The first thing to realize is that we are dealing with infinite solutions. If we can find a single solution (you can imagine a 2D city map), by repeatedly performing arbitrary rotations / reflections / translations (so-called Euclidean transformations) on the city map, we change the coordinates while preserving the distances of each pair of cities and thus we never run out of novel solutions.

The easy case: three cities

Let’s try generating a solution for three cities. Given cities {A, B, C} and distances d(A, B), d(B, C), d(A,C) it’s easy to notice that a valid solution is the set of vertex coordinates of a triangle with edge lengths corresponding to the distances. You can use a ruler and compass: any line segment A̅B̅ of length d(A, B) generates the first two vertices and the intersection of two arcs centered at A and B with corresponding radiuses d(A, C) and d(B, C) will form the vertex C.

The computational approach for the 3 city problem would work like this: start with A=(0, 0) and B=(0, d(A,B)). The compass intersaction is a system of two Pythagorean theorem equations (origin to C and B to C) and with a few rearrangements we can compute a solution for C (note: flipping the sign on y works too - it’s an example of vertical reflection):

The general case: distance and Gram matrices

What about thousands of cities? We need an N×N matrix where each row is composed of the distances of all pairs of cities, but note that we include the distance from each city to itself too (which is obviously zero):

NxN matrices are called square matrices because count(rows) = count(columns). The zeroes are the distance-to-self entries and they form the diagonal of the matrix. Since we are working with a square matrix of Euclidean distances, we want to check out Euclidean distance matrices. The only nitpick is that Euclidean matrices are supposed to have entries that are distances squared to simplify operations that revolve around the Pythagorean theorem. That’s fine - we can square each entry. Note the symmetry between an entry (i, j) and an entry (j, i): this comes from the fact that the distance from A to B = the distance from B to A.

So now we represent the distances in an Euclidean distance matrix D, but how do we represent the solution — 2D coordinates of each city? We can make an N×2 matrix (call it S) where each row represents the x and y coordinates of each city (i.e. a 2D vector). There isn’t a unique way to map D into S because as we said earlier, there are infinitely many solutions. The dimensions of the input D collapse during the transformation from N×N to N×2 - this is an example of dimensionality reduction.

Relation to Gram matrices & dot products

Now, there are different ways of doing dimensionality reduction, depending on which geometric/statistical property you want to preserve when you reduce the dimensions. The properties mentioned in the Wiki article for Euclidean distance matrices such as symmetry and zero diagonals are useless when it comes to coordinates (our solution matrix isn’t a square matrix at all).

But scroll a bit down, and there’s one striking property, Euclidean distance matrices are related to Gram matrices: square matrices that are describing each pairwise dot product of a set of vectors. A dot product v₁ ⋅ v2 is summing the pairwise products of all their coordinates: v1 ⋅ v2 = v1ₓ × v2ₓ + v1ᵧ × v2ᵧ. We can easily see that a dot product of v to itself v ⋅ v is x² + y². That’s the length of v squared (let’s also use ||v|| to represent the length of the vector, also called magnitude).

The geometric interpretation is that v1 ⋅ v2 = ||v1|| × ||v2|| × cosθ. So the entries entries of a Gram matrix G are of form g(i, j) = ||v_i|| × ||v_j|| × cos(θ), where θ is the angle between v_i and v_j. Thus, by storing pairwise dot products, we indirectly store the length and angle of each vector.

The proven relation between D=d(i, j)² and G=g(i, j) is quite trivial addition and substraction:

Turning a Gram matrix into a set of points

The “Finding a vector realization” section further down is what we need, but it’s a bit convoluted since it also considers complex vectors. In any case, it says several decomposition techniques of G can produce the N×2 final result.

Now, explaining matrix decomposition techniques and performing them by hand will require a lot of practice with matrix multiplication, transposes, eigenvectors, but fortunately we can use any numeric computing language/library to do it for us. I chose Sage (which can execute on Web) and SVD decomposition:

N = 4

# note 1: Sage indices are 0..N-1 instead of 1..N
# so we gotta be careful.

# distance matrix.
# note 2: the given distances are not squared
D = matrix([[ 0, 3, 10, 1 ],
            [ 3, 0, 5, 4 ],
            [ 10, 5, 0, 11 ],
            [ 1, 4, 11, 0 ]
          ])

# calculate Gram matrix G as follows:
# g_ij = 1/2 * ( (d_0i)^2 + (d_0j)^2 - (d_ij)^2)

G = matrix(N, N, lambda i, j: (D[0, i] ** 2 + D[0, j] ** 2 - D[i, j] ** 2) / 2)

# SVD of a Gram matrix gives us U * S * U^t
# a possible solution is the product U * √S,
# where the x, y coords of each row are the first two columns
 
U, S, UT = matrix(RDF, G).SVD()

(U * S.apply_map(lambda el: sqrt(el))).matrix_from_columns([0, 1])

If you run it on https://sagecell.sagemath.org/ you get a possible solution:

[                0.0                 0.0]
[  -3.92291410062551   2.528288457959308]
[-10.051441221192283 -1.0157568878859065]
[  0.965029794343825 -0.3021276784799517]

You can check whether this is a valid solution by computing the distances. I got +-15% inaccuracies; presumably because of the square roots and SVD approximations.