Without a 2D polygon describing terrain extent, 3dfier is not able to create a terrain from a point cloud without. The solution is to create a polygon of the area of interest save this as a shapefile.
We make sure the terrain and buildings are stitched nicely and the buildings do not intersect with the terrain. We do this by using symetrical difference between the terrain polygon and the buildings. For nice tutorial about this see section D. Symmetrical Difference at GrindGIS.
Terrain

2D terrain + buildings

Symmetrical Difference operation result

3D model as OBJ in Meshlab

Viewpoint below the 3D model

Thanks to @antoinebio for the images supplied in issue #48.