This site has a lot of informational stuff and usually where I tend to buy things from. They have starter kits available too: http://www.northernbrewer.com/

Another good resource (and if you become a member opens you up to discounts at most breweries) is the american home brewers association: http://www.homebrewersassociation.org/

Like was stated above, start with extract brewing as it will be the least expensive option up front to decide if you want to continue with home brewing. As long as you have a large enough pot (can hold 2 1/2 gallons of water with a bag full of grain at constant boil), you will probably only have to buy:

Primary carboy
Secondary carboy
Air lock
Extract kit with yeast
Chiller
Hydrometer
Thermometer
Cleaning solution
Sanitizing solution
Plastic food safe bucket
Funnel
Bottles, caps and capper (you can reuse old bottles from beer you drink they just have to be sanitized beforehand)

Heating all of that water on your stove top will be hard on your burners, so if you can spare the money I would get an external burner that hooks up to a normal propane tank and get a stainless pot to brew in.

Extract brewing is a relatively simple chemistry experiment. As long as you can follow directions, keep track of timing, and keep your equipment clean/sterile you should be fine. My dad and brother in law have been home brewing for quite a while so I skipped bottling and went straight to kegging as I have never enjoyed having the sludge at the bottom of a bottles. It's not a problem as long as you pour the beer properly. I also liked the advantage to force carbonating beer in a keg so you can essentially have beer on tap in 4 weeks from the day of brewing whereas bottling will add another couple weeks to that.

I've got about a grand into my setup for extract brewing (most of which is tied up in my keezer build for my two 5 gallon kegs and taps). After you have the main equipment, brewing becomes a relatively cheap hobby as you only buy the beer kits.