This is by far the most stressful and confusing thing for beginners (sometimes more than buying a camera). If you're just starting out, choosing the right lens can be daunting...especially when you're investing hundreds or thousands of dollars on what the sales guy told you. But it's not as complicated as it seems, and getting a good understanding of what's driving the prices will help you become a more knowledgeable consumer.
Before we start though, rest assured that lenses hold their value pretty well, so even if you do make a mistake and buy something you don't need, you can usually get a good amount of your money back.
Ok...so what's the key? The vast differences in price really boils down to two things -- manufacturing cost & customer demand.
Customer Demand: What we're talking about here is mostly low light performance. Because, of course, without light you don't get a picture. The ability to shoot in poor lighting conditions is a huge deal for photographers. Even if you don't have perfect light, you'd still like to get the picture right? So things like large maximum apertures so you can let in a lot of light and image stabilization (IS) to help you avoid camera shake are in high demand and drive up prices.
Manufacturing Costs: This goes pretty much hand-in-hand with customer demand. To add image stabilization to a lens, they have to engineer the motor to fit in with the elements while still keeping the lens lightweight and relatively compact. That takes more R&D and more raw materials (like glass) in the manufacturing process to build larger maximum apertures or image stabilization.
Let's look at a couple examples. Below are 2 lenses with a very wide price range.
The biggest difference here is getting that f/2.8 at 300mm. If you look at the first lens, 75-300mm f/4-5.6, you'll notice the front element is the same size as the rest of the lens, and the construction is pretty basic. Moving toward the nice 300mm f/2.8 you lose the zoom, so why is it so much more expensive? It's the construction and size difference. In order for the camera to let in enough light to give a maximum aperture of f/2.8 at 300mm it needs a HUGE front glass element to gather as much light as it can. That way, by the time the light reaches the camera there's still a lot left.
Take a look at another size comparison from camerasize.com. Both lenses are mounted on a Canon 80D (standard DSLR camera body). On the bottom is another basic kit lens, an 18-55mm f/3.5-5.6. And on the top, the same 300mm f/2.8. Now can you see the difference in size? The larger they have to make the glass elements the faster that price goes up.
Let's look at one more example. This one is a little more realistic since many of you won't be dropping over 6G's on a lens anytime soon. Below is the comparison between Canon's different 70-200mm lenses. On the left is the 70-200mm f/2.8 and on the right is the 70-200mm f/4. You can clearly see the large difference is size from f/4 - f/2.8. Bigger glass = bigger price tag. The basic f/4 is priced at $600, and the f/2.8 IS is priced at almost $2,000.
While there's still other factors that go into a lens pricing, hopefully this gives you a little insight into why they can be SO different. It really boils down to, every photographer wants to be able to shoot in less-than-favorable light -- Customer Demand; and at the same time the cost to build a lens like that increases -- Manufacturing costs.