I have come to believe that supplements have the greatest effect when they are correcting for deficiencies. If your body is lacking in the things it needs to perform, it makes sense that providing it those things would be productive. There are a number of ways your body can end up lacking in things it needs, primarily through lack of dietary intake/exposure, or through processing anomalies that cause some people to absorb and process things more or less efficiently than others. Eating fresh, seasonal produce and a variety of wild or grass-fed/pastured animal products, and getting natural sunshine is the best way I know to protect against deficiencies. Supplementation is another option, but targeting the intake of specific micronutrients, especially fat-based Vitamins A, D, E, and K, or minerals like Iron, can lead to toxicity, so is best done under the supervision of a doctor.
Don't use them. Don't use supplements that might be contaminated with them off label. While I believe there are persuasive cases to be made that the line around PEDs has been drawn in the wrong place, or that the rules should be more or less lenient, the line has been drawn where it is (you can learn about what is considered a PED on USADA or WADA's websites), and while it is partially drawn to even the playing field, I believe it is more so drawn to protect athletes during their competitive careers, and in their lives after sport. Some of the negative short- and long-term consequences of drug use have been publicized, others are not known. Mostly, I believe that for people who are pursuing optimal health, PEDs have no place. They may be performance enhancing, but they are dangerous, unethical, illegal, and in short, a really bad choice even if they work for their intended purpose.
Once you've established sufficient presence of protein for muscle growth and recovery (unless you're trying to lose weight, see BCAA's below), and carbs for muscle fuel, Creatine and HMB, especially used in combination, are believed to be effective in promoting performance in the phosphocreatine pathway, aka the immediate energy system. Beta Alanine is effective at improving stamina in the Glycolitic pathway, or efforts of up to about two minutes.
Creatine is a substance naturally occurring in meat which is a substrate for the formation of ATP within a muscle without using oxygen. The theory is if you don't have enough creatine in your body this system will be less effective in 1RM and short sprint efforts. Despite some negative initial research published in my formative years, it is generally believed to be safe and effective at a 5g/day intake, except in those with pre-existing conditions (like kidney disease). The primary side effect is weight gain - through muscle #gainz and water retention. The former is real and sticks with you, the latter goes away in a few days once you stop taking it.
The branched chain amino acids (BCAAs) are Leucine, Isoleucine and Valine, three essential amino acids which are used to build muscle. The theory with BCAAs is that you can provide your body with these specific muscle components without consuming the calories associated with a full protein source. This is a supplement to consider if you're trying to lose weight while maintaining or gaining strength (beware that most BCAA supplements contain artificial sweeteners because they're super bitter on their own).
HMB is a metabolite of leucine, which is one of the branched-chain amino acids (BCAAs). The theory is that this compound helps to reduce muscle tissue breakdown even more than leucine in it's original form.
Beta Alanine is a metabolite of alanine, which is also an amino acid, but not a BCAA (so confusing). It becomes carnosine in the body, and is useful in improving muscular and anaerobic endurance, while decreasing fatigue and possibly helping to reduce fat mass. This is the ingredient in pre-workout supplements which can cause tingling called paresthesia (which is supposedly harmless). If you don't like this feeling, you may want to introduce this supplement slowly.
(If you are science minded and see anything in the above that doesn't jive with your understanding, please let me know!)
Primarily in this category (see above Deficiencies & Toxicity), it's somewhat risky to target the specific intake of a particular vitamin or mineral due to the risk of overdose, and it may be a waste of money given some theories that supplements, especially in synthetic form, aren't very well absorbed by the body. That said, a few supplements are worth considering:
Fish Oil - if you don't eat fish 3-5 times per week, it's possible you aren't eating enough Omega 3's (fatty acid). Omega 3's are useful for heart health and reducing systemic inflammation, especially in diets also high in Omega 6's. Fish oil is a blood thinner at high dose, so be cautious if you're preparing for surgery.
Magnesium - due to overfarming, produce doesn't incorporate as much magnesium from the soil as it used to, reducing our intake as a society. It's helpful in muscle recovery, and also needed for most of the body's processes. It's also a laxative at high dose in some forms, so heed the warnings on the label to introduce this supplement slowly.
Vitamin D - this vitamin is naturally produced by the skin in sunlight, which is tricky in a world filled with fears of skin cancer, or limited by the San Francisco fog. Deficiency is likely, but should be assessed and monitored through blood tests under the care of a physician.
Glucosamin Chondroitin - if you don't eat connective tissue or Jello you may not be consuming enough collagen/gelatin to support connective tissue (cartilage/ligament) recovery. There are mixed reviews regarding this supplement's effectiveness, but I have had good results personally with Cosamin DS.
Turmeric (typically combined with curcumin and black pepper extract for maximum effectiveness) - this spice acts as a natural anti-inflammatory, and can be used instead of NSAIDs (Ibuprofen) to assist the body in reducing swelling post workout. This shouldn't be much of an issue when you're eating Paleo, as it's an anti-inflammatory style of eating, but in those cases where you're sooooooo sore, or a joint is a little flared up it can help. That said, it's a supplement, not a lifestyle, so if you need it all the time, figure out how to adjust your training and recovery program in order to not need it so much.