This blog post How can I return a 503 status in apache without invoking external scripts basically explains it all, but I had to struggle a bit before I completely understood the needed arguments. So now I have the following lines at the end of the httpd.conf file (not for this domain but a server at work 🙂 ).
Redirect 503 /test503
Redirect 404 /test404
Redirect 501 /test501
Redirect 504 /test504
Redirect 505 /test505
Redirect 502 /test502
Redirect 506 /test506
Note that you have to make sure that mod_alias is enabled