|Cute caveman image found here|
What can you eat? The diet consists of only eating foods that we ate when we were cavemen/hunter-gatherers. That includes fruits and vegetables, meats, eggs, fish, nuts and seeds, raw honey
What can't you eat? Anything that is not a food (i.e. junk food, candy, packaged foods), grains (including gluten free grains, oats, and pseudo grains like buckwheat and quinoa), dairy, legumes, sugar, oils
Can you bake on this diet? Yes. There are lots of Paleo cookbooks and blogs devoted to baking breads, muffins, and cakes, only they use coconut flour and almond flour (which are both grain-free), and eggs to create a "dough". I've tried making paleo bread, brownies, and cake, all with success (I like to be challenged in the kitchen and paleo baking provides me with some challenge!). So yes, baking (and thus eating sweets) is definitely still possible.
As with all diets, there are some extremes and exceptions...
Exceptions: Generally, this diet is high protein, low carbohydrate, and it eliminates grains. As with any diet, there are many variations and exceptions. The only "rule" that seems to remain consist between all of the variations is no grains as grains are a "new food" to humans that was invented recently during the agricultural revolution. Other than that, there are people who are paleo but still eat chocolate, sugar, cheese, and sausages (i.e. foods that caveman certainly didn't eat, but at least they contain any grains). On the other hand, there are others really stick to eating primal foods, which means mostly fruit and vegetables that can be gathered and animals that can be hunted.
Extremes: Some people who follow the paleo diet go as far as living like a caveman, meaning they don't wear shoes (they will wear Vibrams instead), won't eat bacon (pork is paleo, but our ancestors didn't eat bacon on a regular basis, it a new way of preparing pork), perform regular fasts (animals weren't able to be hunted everyday, hence the fasts in between kills), and exercise like a caveman (sprint, lift heavy objects, climb trees, etc. ...because there were no elliptical machines when we were cavemen!).
Some diet Pros and Cons...
Pros: Less junk/fake-foods, lower in carbohydrates (especially simple, refined sugars, it could be a good diet for diabetics?), a very clean diet (no package foods with artificial stuff allowed), naturally gluten-free, can include lots of vegetables and fruit, high protein (especially good if you are an athlete), apparently by following the paleo diet you'll have a better omega 3 to omega 6 ratio (don't have the source for this, but heard it in class last week), lots of online support communities
Cons: No research (longitudinal studies especially) supporting the paleo community claims that by following this diet you will reduce your chances of acquiring Western diseases, eating lots of meat can be hard to digest, can be expensive (almond flour and coconut flour are very expensive compared to grain flours like whole wheat or buckwheat), considered a "fad diet", ranked poorly among dietitians (says the Wikipedia article on Paleo, ha ha), not environmentally sustainable, can get boring eating meat and veggies only, like a modified Atkins Diet (which has been shown to harm people's health), many people abusing the diet to just eat bacon and steak all day long :P