The moduli space of semistable Higgs bundles of arbitrary rank and degree on a nonsingular projective curve was first constructed by Nitsure in 1990, using Geometric Invariant Theory (GIT). Thanks to its rich geometric structure, this moduli space continues to represent an active area of research. The aim of this talk is to describe how recent results in Non-Reductive GIT can be used to construct moduli spaces for Higgs bundles which are not semistable, and to describe initial steps towards the study of their geometry in the rank 2 case. In the first part of the talk we will start by giving a summary of Nitsure's GIT construction of the moduli space and describing the main geometric features of the moduli space. We will then consider the special case of (twisted) Higgs bundles over the projective line, in order to introduce unstable Higgs bundles and their moduli spaces in an elementary way. In the second part of the talk we will sketch the Non-Reductive GIT construction of moduli spaces for unstable Higgs bundles over a smooth projective curve of arbitrary genus. We will then describe how the geometry of these moduli spaces can be studied in the rank 2 case, using the Higgs field scaling C-star action on the one hand, and their construction as Non-Reductive GIT quotients on the other.