House of Leaves, Mark Z. Danielewski. Usually on the general fiction shelves.
Not "scary" as in a lameass movie that just startles you, but it is very interesting if you have an imagination and the self-control not to google or skip forward pages
Drood, Dan Simmons. Again not "scary", but very unique.