Chad and IPRO offered excellent responses. Thumbs up to those guys. I will share a bit more detail.
Prior to hiring an SEO, you need to clearly understand SEO is a long-term approach to building your business. If you desire instant results, you can do Pay Per Click advertising. It will cost a lot more but if you desire instant results and maximum control, PPC is the way to go. I say that being an "SEO guy". Prior to hiring a SEO, it's important you have reasonable expectations otherwise you are going to be disappointed.
Also, I would recommend not hiring your SEO company for 60 days after your initial contact. Why? It gives you time to consider other companies, process the sales pitch given, and perhaps measure some results or commitments.
These are the questions I would ask if I were hiring an SEO company:
1. **Who exactly will be doing the SEO on my site? **You should receive a name, a phone number along with their social accounts and instant messaging (skype, etc). Why is this important?
a. As suggested by others, many SEO companies range from shady to pure scams. Some companies are simply fronts who accept business for many services and outsource everything. They do not even know or have any idea of who will be performing the SEO for your site.
b. A ton of SEO work is outsourced to India or other countries where the "SEO" assigned to your account is working for $2/hr. I am not suggesting there is anything wrong with this practice other then the lack of transparency. Buyers should know exactly what they are purchasing.
c. Over the 60 day period, you may wish to call the person a few times and ask general questions. Does the person answer their phone sometimes? Or do your calls always go to voice mail? Does the person respond to e-mails promptly? Ensure the SEO working on your site is someone you feel comfortable working with.
d. Research the SEO. You can check their standings in various communities, their credentials, etc.
e. Interview the SEO (not the company but the actual SEO). Ask questions related to your site.
All of the above direct research is very real and cannot easily be manipulated. Many of the references offered by SEO companies range from misleading to completely false. I worked for Verizon Wireless for 7 years. As I visit various websites I am surprised to see so many companies claim Verizon Wireless and other large companies are their client or use their tools. Upon investigation I learned either the claim is completely false, or it is misleading. Perhaps "Joe's Cellular" has a single store and is a Verizon Wireless authorized retailer. Joe approaches a company for SEO work and then the company claims Verizon Wireless is a client. It is very misleading.
With the above noted, it is also easy to fake references as well. Since IPROdigital mentioned the "TOP SEOs" site, I'll use that as an example. That site has been called out multiple times as being a sham. Any SEO can go to the site, pay money, and then be listed as a "top seo" in a given area.
2. What is the background of the actual company offering SEO services? Many websites give the appearance of representing a large company, but in actuality there is a single person running the site out of their home. There is nothing wrong with working directly with a lone consultant. I performed business in that manner for a period of time. But you, as the client, should know. Also, there are many companies in India and other countries who perform 100% of their SEO work in India, but they will open a virtual office in the US, thereby claiming to be a US company. Some suggestions:
a. Check for a business license. Anyone can open a website. More transparency is required to open a business. Most areas publish business registrations online. Ideally your company will be some form of corporation. In the US, you can check with the Department of State. Example: http://kepler.sos.ca.gov/. For a test you can choose the LLC option then type my company name, "Vitopian". The company should likely have a license for their local office registered with their county or city.
b. Check with the Better Business Bureau. A business needs to exist for 1 year before the BBB will accept registration. http://www.bbb.org/us/Find-Business-Reviews/. There is nothing wrong with not being registered with the BBB, but registration is an indicator the business complies with certain standards and is actively open to mediating complaints.
c. Check with DUNS and BRADSTREET: http://www.dandb.com/businessdirectory/. The US government requires businesses to use their DUNS number in order to bid on government contracts. DUNS is the key credit reporting number for businesses. Many businesses do not even realize they have a DUNS number.
d. Check any industry groups to see if your SEO is a member. One example would be SEMPO: http://www.sempo.org/?page=member_directory
To be clear, the above information needs to be considered as a whole with the other information you collect. A business could be a fantastic web design agency which has all the right credentials, but has limited experience and knowledge related to SEO. Checking the above items will help filter out a lot of low quality companies, but passing this "test" does not mean you have made a good choice for a SEO agency.
3.** What has your SEO agency done lately**? Many companies performed well pre-2012 and are absolutely bombing in 2012. If you were looking for a stock broker, would you hire one who showed great results 2000 - 2010 but had a horrible 2012? Google and Bing have both made huge changes recently. Panda and Penguin are game changers. A few companies have been able to adapt, while most have not. You want to know who can help you in the current SEO environment, not that of years past.
4. Ask for a high level SEO plan. Most agencies will tell you they intend to build links to your site. My advice...run! At a high level, SEO comprises three elements: website architecture / design, content, and site promotion (social media + earning links). Once you have a solid website with great design and fantastic content THEN you can begin promoting the site.
Many SEOs are still caught up in years past where you could simply build links to a site and get it to rank well. They never learned about onpage optimization beyond adjusting a few meta tags. In my experience the best long term results are achieved for a new client when you first focus their website architecture, then work to raise the quality of the site's existing content to "best on the web" levels. You need to distinguish between "link builders" and SEOs. A working analogy is the difference between a guy who changes oil and a certified mechanic. There are too many oil changers who are presenting themselves as mechanics.
5. Insist on full transparency. If your car was not performing as well as it should and your mechanic said "we will use our secret process to improve your performance", would you buy that line? In 90%+ of cases, "secret", "proprietary" and similar terms is slang for "black hat". Here's a suggestion. If someone offers you such a line, ask for the information in writing. If you are speaking on the phone, ask if you can record the call.
6. Consider the SEO providers commitment requirements. What happens if you sign a 1 year contract and then are unhappy after 1 month? What happens if you pay a large upfront fee for work and then are unhappy? In either case, you are pretty much screwed. If you find an agency who does not require a term contract nor large upfront fees, then you can walk away at any time.
7. Consider the SEO's availability. I loved IPROs statement "We're booked beyond capacity..." Many of the best SEO agencies are completely booked. Growing the business in a quality way takes a LOT of time. It takes months to train a recent college graduate into becoming a productive member of a SEO team. Hiring existing talent is hard because many "SEOs" are simply link builders or use black hat techniques.
There is a lot more areas which can be discussed but ultimately you need to chose an SEO in the same way as you would consider hiring an employee to perform SEO. You need to consider your budget and your expectations. If you have a low budget and high requirements, you are setting yourself up for failure. You need to invest some time considering several providers, avoid any high pressure sales pitches and seeing through presentations to determine what is meaningful for you.
A final note. There are many metrics related to SEO. Ranking, PR, traffic and conversion rate are just a few. Here's the thing. For most business owners, none of those metrics matter. Each of those metrics can be manipulated. The truest measure of performance for a business owner is profit. Another metric is sales. Keep that in mind when discussing other metrics.