Laplace's Demon and the Singularity
Introduction:
In my first year as a math undergraduate I remember a fasttalking physics student who stated that if you folded a piece of rectangular paper enough times its length would cover the distance between the Earth and the Moon. Disbelieving, I thought for a few minutes and countered that given that paper is inelastic the maximum length can’t exceed the length of the diagonal. Nevertheless, that year I discovered that this belief was remarkably popular among physics students whose chief interest was to imitate Richard Feynman.
A similar belief, known as the technological singularity, possesses the minds of many AI researchers although its flaws are numerous and would be obvious to an outsider with a basic understanding of science. If I may sum it up succinctly, the singularity is the idea that at some point in the future AI engineers will create a super human intelligence which will engineer exponentially smarter versions of itself so humans would no longer have to do science anymore.
In the following article I argue that an irremediable flaw in the singularity notion is equivalent to the problem of Laplace’s Demon which has been addressed by statistical physicists in the past.
Main points:

Entities whose ‘intelligence’ is monotonically increasing due to recursive selfevaluation converge to an allknowing ‘superintelligence’:
If I take ‘intelligence’ to mean a proxy measure of an agent’s degree of control over its environment, aka Empowerment [1], then the above statement is equivalent to a theory of everything we know and everything we don’t know. You would be literally extrapolating beyond what could be reasonably justified. From this it follows that the above statement is plausible with vanishing probability. I shall clarify this statement below.

Assuming discrete time steps, learning time is an exponential function of the median planning horizon:
In a stationary environment which has sparsely distributed spatiotemporal reward signals the median planning horizon will tend to be quite large. Let’s suppose that by large we mean 10 time steps into the future and that the agent has a discrete action space of size 4. In this case, the agent would need action sequences in order to discover an optimal policy. We haven’t even considered the case of survivor bias: adversarial action sequences where the agent regularly obtains a reward due to chance but can’t reliably obtain this reward in the future due to a sequence of actions. Furthermore, the case of a continuous action space isn’t very different because you can construct an equivalence relation over actions which produce the same outcome or goal within time steps.
What this means is that without a good model of the environment, higherlevel abstractions which would allow the agent to reduce the learning time required, are nonexistent. My next point reiterates this.

Assuming the existence of atomic actions, learning time is an exponential function of the degree of hierarchy/abstraction of the action space:
In 1814, none other than Pierre Simon Laplace argued that we could theoretically model all behaviour with some large Newtonian manybody system. The exact quote is the following:
We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed, if this intellect were also vast enough to submit these data to analysis, it would embrace in a single formula the movements of the greatest bodies of the universe and those of the tiniest atom; for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes.Laplace
Assuming Laplace is correct, if we make minimal assumptions about the environment, how long would it take for complex locomotion behaviours to emerge from this system? Closer scrutiny of this question would reveal that it’s essentially equivalent to the previous question. Assuming actions at the microscopic scale(ex. muscle activations), complex locomotion behaviours are equivalent to very long planning horizons. Without good prior knowledge(i.e. a model) of the hierarchical/compositional behaviour of muscle tissues you might as well be running a simulation for ages in order to obtain the appropriate samples.
The situation is actually worse than we suppose as Laplace is incorrect in the case of the observable Universe. Physicists since Boltzmann have demonstrated that information is lost over time so besides the combinatorial complexity of simulation there’s the issue of uncertainty propagation across the simulation with every calculation.
Summary:
For the reasons I gave above I think it’s clear that in complex environments of which nothing is known, any reasonable interpretation of superintelligence(ex. maximal empowerment) is essentially equivalent to Laplace’s Demon. It follows that the asymptotic limit of an allknowing superintelligence is not only highly improbable. Given our current scientific knowledge, it’s practically impossible.
Singularitarians forget that science is about quantifying what we don’t know via experiments and not attaining ‘humanlevel’ understanding. Barring omniscient robots, I don’t see a point in the future when human scientists will be out of a job.
References:
 Salge, Calkin & Polani. Empowerment – an Introduction. 2013.
 Laplace. A philosophical essay on probabilities. 1814.