This is MY town!
That said, skip Fisherman's Wharf - it is a serious tourist trap. You won't see much other than silly tschotsckies and chain stores. The food sucks as well.
Are you a foodie person? I was a Food & Wine writer so I can make LOTS of suggestions where to eat and where to stay away from. A big thing to do in the city is on Saturday mornings: The Ferry Plaza. It is our Farmer's Market but so much more -- it is the place to be and be seen in the city and if you start there in the morning, you will be fine for breakfast and lunch. There are tea shops, oyster bars, bakeries, chocolatiers (Rechiuti - way better than Ghirardeli), olive oil stands, etc.
Yarn-wise, I'm a big fan of Atelier on Divisidero. It is not necessarily near anything specific, but if you are thinking of visiting Japantown (which is in the center of town), it is about a 10-minute walk from there.
Besides your standard tourist locales (Alcatraz, the Wharf, Golden Gate Bridge) San Francisco has some great jazz clubs. I'm a fan of Biscuit & Blues which is very near Union Square -- they serve gumbo and other great southern dishes along with the music.
If you are a movie fan, there is a 3-hour bus tour that is really cool (although slightly expensive at $50 a person). The San Francisco Movie Tour drives you around the city, showing clips of famous movies and where they are located within the city. I took my BF's mother last Thanksgiving and we both loved it. You actually watch scenes from flicks like 'Vertigo' or 'Dirty Harry' and then they show you where it was filmed.
Union Square is shopping central - Neiman Marcus, a 5-story Macy's, Saks, Tiffany, Borders, etc. But it is also central to several streets worth of art galleries, my current obsession. You can see (and purchase) Miro, Picasso, Renoir, Donati, Rembrandt, and more without paying museum-entrance fees.
Golden Gate Park is huge and rivals New York's Central Park for mass. Within it are arboretums, museums, golf courses, concerts, a butterfly exhibit, and just plain gorgeous botanicals. I would see if your concierge could arrange rent-a-bikes for you if you are nature lover.
Chinatown is very interesting and can get very crowded. My favorite store there is The Wok Shop where they sell the best cooking supplies.
Tell me what you like to eat and I'll suggest places to dine. San Francisco is the culinary mecca of the U.S. with arguably the best dining in the country. Whatever you do, stay away from chain restaurants (no Dennys or TGIFridays!) as you can do so much better during your stay.