With 13.10 (albeit the desktop version) I found that I had to build PostGIS 2.x from source, as the ubuntugis-unstable PPA does not include the *.sql scripts necessary to register the extension in and populate a template database (I think mattmakesmaps alluded to this in his answer). Here are the basic steps for building PostGIS 2.1 from source on top of PostgreSQL 9.1. This approach assumes that you installed PostgreSQL 9.1 from the package (postgresql-9.1) and also installed GEOS, Proj, and GDAL form source or can otherwise find the shared libraries.
# Install dependencies
sudo apt-get install libxml2-dev
cd /usr/local/ && sudo mkdir postgis && sudo chown [username] postgis && cd postgis
wget http://download.osgeo.org/postgis/source/postgis-2.1.0.tar.gz
tar -xzvf postgis-2.1.0.tar.gz && rm postgis-2.1.0.tar.gz && cd postgis-2.1.0
sudo ./configure --with-projdir=/usr/local/proj/proj-4.8.0/
make
sudo make install
# Configure template database
POSTGIS_TEMPLATE=postgis-2.1.0
POSTGRESQL_VER=9.1
sudo su -c "createdb $POSTGIS_TEMPLATE" - postgres
sudo -u postgres psql -d postgres -c "UPDATE pg_database SET datistemplate='true' WHERE datname='$POSTGIS_TEMPLATE';"
sudo -u postgres psql -d $POSTGIS_TEMPLATE -f /usr/share/postgresql/$POSTGRESQL_VER/contrib/postgis-2.1/postgis.sql
sudo -u postgres psql -d $POSTGIS_TEMPLATE -c "GRANT ALL ON geometry_columns TO PUBLIC;"
sudo -u postgres psql -d $POSTGIS_TEMPLATE -c "GRANT SELECT ON spatial_ref_sys TO PUBLIC;"
If you want to install shp2pgsql you'll need to do the following.
# Put the loader `shp2pgsql` on your path; likely not done when building from source
cd /usr/local/postgis/postgis-2.1.0/loader
make
sudo make install
sudo ln -s /usr/local/postgis/postgis-2.1.0/loader/shp2pgsql /usr/local/bin/shp2pgsql