One of the final frontiers in modern cosmology is the formation of the first stars and galaxies at the end of the cosmic dark ages, when the Universe transitioned from its simple initial state to one of ever increasing complexity. I will discuss how the first bound gas clouds condensed out of the smallest dark matter halos, ultimately leading to the formation of the first stars. State of the art numerical simulations are our method of choice, showing that their formation is similarly complex to star formation today. Their radiative, mechanical and chemical feedback influences the formation of the first galaxies, and results in a highly turbulent and multi-phase interstellar medium accompanying the second step of hierarchical structure formation. These objects are the targets of the next generation of ground and space-based telescopes, which can probe the Universe when it was only a few hundred million years old.