Hi everyone,

With so many people asking for feedback, I thought it would be great if we had a good system of receiving feedback on good and bad companies without having to hunt through pages of postings.

I came up with an idea I'm calling "MSC Scorecard", but want to see how many would be interested. I know that many of you don't like to post on the boards, but though you may be more involved in offering feedback if it was anonymous.

To get started, I'm collecting info on the BEST and WORST companies of 2009. I will publish the results here at the end of the year for all to see.


--The survey is now closed--


Thanks for your input. Please check my post from 1/1/10 for the results.

Edited 1 time(s). Last edit at 01/01/2010 05:48AM by SteveSoCal.

OK, while I'm really posting to bump this thread back up to the top (it got knocked way down by the spam attack shortly after I first posted), I wanted to share that I find it funny that some of the same MSCs have been nominated for both BEST and WORST to work for.

If you haven't done so yet, please take two minutes and follow the link above to the survey, and add your feedback.

I think that would show a disparity in level of experience....does it?

I just went and took the survey. I find it interesting that I had to add favorite and least favorite msc's to the list. That makes me curious about the list. How did you derive it, Steve?

EDITED TO ADD:

dumb me. I just saw you used the list here. Apparently it needs some updating...I wonder who maintains the list.



Edited 1 time(s). Last edit at 12/20/2009 11:50PM by dee shops.

Yeah, the list will need updating if there is any intent on going forward with an actual rating system. One of many hurdles to overcome.

I don't collect any shopper data for this because I wanted to keep it simple, but experience level would be a good piece of data to collect in the future.


I was also thinking about how to deal with bad companies that have made good, or the contrary; I think the system would have to function like a credit rating, where good or bad data would expire over time and be replaced by current data.

dee shops Wrote:
-------------------------------------------------------
> I think that would show a disparity in level of
> experience....does it?

It could also have to do with different schedulers and the amount of shops in an area.

lisams901 Wrote:
-------------------------------------------------------
> dee shops Wrote:
> --------------------------------------------------
> -----
> > I think that would show a disparity in level of
> > experience....does it?
>
> It could also have to do with different schedulers
> and the amount of shops in an area.


Yes, definitely.

great idea... sort of an angies list for mscs... this is really awesome... go for it...

I'm still taking submissions for the 'MSC scorecard' survey until midnight PST on Thursday!

For those that were asking, the survey only allows one entry per IP address, so anyone trying to vote twice for their favorite company will have to borrow their neighbor's wi-fi signal!

Optimistic to think we would borrow an IP address to elect a favorite? Passions may run higher for the least favorite . . .

Mert Wrote:
-------------------------------------------------------
> Optimistic to think we would borrow an IP address
> to elect a favorite? Passions may run higher for
> the least favorite . . .


LOL.

I agree Mert.

The results are in for the best and worst mystery shopping companies to work for in 2009.

Some of the results are expected and others were educational for me. You may not agree with the results, but that's the data I received.

Here’s how it worked; Rather than scoring companies on overall votes, since some companies are larger and have more shoppers, the overall company score has been used to justify placement. Scores have been averaged from 10-point rating scales, and rounded to one decimal place, based on the following elements:

• Frequency/Timeliness of Pay:
• Pay Rate:
• Quality of Assignments:
• Quality of Instructions:
• Quality of Reporting:
• Scheduling:
• Communication with MSC:
• Resolution of Problems:

All scores have been verified to come from independent IP addresses. Scorecards that were incomplete, or voted all 10’s for the best and all 1’s for the worst MSCs have not been counted in the totals, since I do not personally know of any companies that extreme. In addition, to be considered for scoring, a MSC must have received a vote by more than one shopper.

Some companies had votes in both categories and therefore scored close to the average (6.9) for all companies. These included:
Sentry (7.1),
Service Intelligence (6.7)
CORI (6.2).

Companies are listed below as: Company (Average Score)

Best Mystery Shopping Companies of 2009

1. Maritz (9.5)
Pay Rate averaged 8.6, and was the lowest scoring component for this MSC. It was the only average score below 9, making Maritz the favorite MSC of 2009.

2. Mintel (9.4)
This MSC had generally high marks overall. Resolution of Problems was the only component that averaged below 9, bringing Mintel in at second place.

3. Trendsource (9.2)
Pay Rate was the largest problem for this popular company and averaged 6.3, bringing the average down despite all other areas scoring well.

4. Coyle Hospitality Group (8.3)
High marks were given for Frequency/Timeliness of Pay, Quality of Assignments and Communication with Coyle, but the overall score was reduced by lower averages on Pay Rate and Scheduling.

5. Market Force (7.2)
Median averages for Pay Rate and Quality of Assignments kept this MSC from placing higher on the list of favorites. It comes in slightly above average for all MSCs.

Overall, MSCs in the top four generally scored between 8 and 10 in most categories, with the exception Pay Rate, where the average score was 8.2 and scoring went as low as a 3. Resolution of Problems was the next lowest category, with a 9.0 average. This can most likely be attributed to a number of ‘No Opinion’ (5) scores from shoppers who have not encountered problems with their favorite company. It appears that although many shoppers are unhappy about the existing rates of pay, it does not prevent them from working for a company or deeming it their favorite.


Worst Mystery Shopping Companies of 2009

1. JC & Associates (3.3)
This company has been notorious for late payments recently and had very low marks for both pay categories and Communication, with average scores for Quality of Assignments, Scheduling and Quality of Reporting. This makes JC & Associates the least favorite MSC to work for in 2009.

2. Feedback Plus (3.6)
Scoring was low overall for Feedback Plus, with Pay Rate being the worst factor.

3. Ath Power (3.7)
Higher averages for Quality of Assignments and Quality of Instructions helped to boost the overall score, but negative scores regarding missing and low pay, as well as Communication and Resolution of Problems kept Ath Power on the list of worst companies.

4. Intellishop (4.1)
The Frequency/Timeliness of Pay was the main factor in bringing Intellishop to this list. Above average marks for Quality of Assignments and Quality of Instructions drove the overall score higher.

5. Bestmark (5.1)
Frequency/Timeliness of Pay was the high point for Bestmark, scoring an average of 8.3 and bringing the company average up from what would have been a 4.6 without it. All other scores were below average, making Bestmark a reliable, but not very profitable to shop for, MSC.

While most categories in the least favorite companies had below than average scores, Quality of Assignments and Quality of Instructions had average scores closer to 6, with Scheduling and Quality of Reporting each averaging in the 4’s. Frequency of Pay and Communication were slightly lower, with Resolution of Problems averaging 3.2, and clearly being a driving factor in the vote for worst MSC of 2009.

None of my favs made the list. Just as well..more shops for me. I note that one of my other favs made the worst list, and for poor pay. Not here. Not for me. I also note that it seems like many (but not all, like Coyle, for example) seem to be ones popular and easy for newbs. I still think this needs a way to validate time in profession...It really makes a difference in scoring some places, as the ones I rated highly are more selective than many others here that are self-select for shops with no screening of potential shoppers. I personally shop a lot or occasionally (or have stopped due to lack of payment for one MSC..) for all mentioned except Ath and then Coyle, which has never accepted me as a shopper. These almost to a number fall in my middle of the road group. I think the survey is a great idea that still needs tweaking to get to the real meat for experienced shoppers. For newbs, it appears to be a very workable foundation. I also note that I gave my highest rated MSC's the highest ratings in all categories-hey, that is why I love them. Why discount that? It is real feedback from an experienced shopper...



Edited 3 time(s). Last edit at 01/01/2010 05:49PM by dee shops.

Yes, I look down the list and it does not correspond to the companies I avoid like plague and the ones that are my favs. Nor does it align with the companies I worked the most for in 2009 and looked forward to working with.

Dee, just to clarify; I only discounted the entries where a shopper put all 10s for their fav, PLUS all ones for the worst company.

To me, that just doesn't ring true. If a company is absolutely terrible in every sense of the word, they would not be in business. These entries were few, but to me they are the mark of someone who had a bad experience with a MSC and just wants to dis them, rather than offer honest insight as to where the problems lie. Most entries were very honest and what I would expect from experienced shoppers (both bad and good points were noted for the problem companies). That's why the average score was well above a 5.

The main thing I learned from reading the entries is there are as many opinions out there for what makes a company good or bad as there are MSCs. There were no clear winners or losers by popular vote, which is why I went with the rating scale (that wasn't my original intention).

Because of the varied opinions out there and the inability of most people to follow simple directions or offer objective data without being paid, I don't think it's feasible to quantify the best or worst companies.

Another survey where I simply ask what the average pay, reimbursement and time to get paid from each company would give more useful results. However, that info is pretty readily available anyway and steps on a lot of ICA issues when asking others to give it up.

I had a thought that perhaps the survey info may be useful for some MSCs to see, but I don't think a letter to JC and Associates letting them know that shoppers are unhappy about pay will teach them anything, or really make a difference.

Happy new year ...

I would put the company PROS in the top 10 of best MS companies.I dd quite a few bank jobs for them,they paid me earlier than the stated pay date.Plus I got excellent feedback on my reports.This is the first company that ever responded so positively to reports I submitted.I was sceptical at first when I did the jobs because I did a route of bank jobs(5)and wasn't sure about the pay,but these folks are a really good company.I also work for Maritz and I agree that they should be # 1.

Of the top 5, I have only shopped with four of those, but those four would be counted as some of my favorites to shop for.

Of the lowest 5, I have shopped with three of those and two of those three I would consider to be in the bottom.

I would like to see the raw ratings, and not the averaged data, I guess. For every MSC that got a score, and how many people voted.

Interesting results. I'd also like to know how many people voted.

Steve, I was thinking about the survey today, and the issues in trying to rate the "best" and the "worst" against an uneven population base (where MSC's have jobs, how experienced or inexperienced they accept shoppers, shoppers comfort with those that required full narrative, etc. I agree with you that I am wondering if it might be more interesting as a Zagat-type guide like someone mentioned. Where all the feedback is published, and just listed alphabetically. Maybe even a wiki? Just a thought. I know you have a job to do and that this is not paid work, so maybe it is just too unmanageable. :-)

I haven't given up on the idea of the Zagat type guide. I'm working on a way to do it free of charge to anyone. It was clear from the comments I received that charging for it won't work.

The survey questions need a bit of tweaking and I want to start collecting shopper data like you were asking about, Dee.

This was a good trial run, so expect a few more surveys and perhaps we'll have something interesting by the end of 2010.

Steve, take into consideration many shoppers are in rural areas where they dont get a chance to shop some companies, and are limited. Maybe doing large cities, then rural areas could help, and have shoppers tell how much experience they have (newbie's excluded), sorry, come back in a year. None of my favorites made the list, I work monthly for many of the same, so, of course they would be my favorites. JC&Assoc. is now on everyones worst list, however, I worked happily for them for over 3 years, until there downfall and problems. They are now Marketstat. Don't know if it's fair to put them on the worst list for 1 bad year, with the failing economy and losing there main client here in L.A. This is also personal, depending on various jobs shoppers like to do. You have your work cut out for you.

I could be wrong and I'm really not sure of what Steve's job is, but I thought he did this just as a little unofficial experiment. Just to get an average basic idea. It seems that the results are being blown way out of proportion.

Everyone is going to have different ideas of who is the best and worst and that is going to be dependent on so many factors. Things that are important to one shopper may not be that important to another shopper therefore what makes one company great to one shopper won't matter to another shopper.

This list is not trying to be passed off as proof of anything and I doubt it is going to have any impact on the companies themselves.

I haven't seen anything blown out of proportion. I think people are trying to make it useful. There is a difference.

dee shops Wrote:
-------------------------------------------------------
> I haven't seen anything blown out of proportion.
> I think people are trying to make it useful.
> There is a difference.


I considered this to be blown out of proportion:

"Maybe doing large cities, then rural areas could help, and have shoppers tell how much experience they have (newbie's excluded), sorry, come back in a year."

Again, I didn't think this little informal survey was meant to be taken as fact or proof of anything. Why would it need to be broken down into geographic location? I find it also a little rude stating how much experience one has to have before they can vote.

I could understand if people were being paid to compile and pass out this information. I could understand if this list was going to be passed out to clients deciding which MSP to choose. But, at least to my knowledge, that is not the case.

Lisa, you are correct that the survey is just informal. It was not meant for anything except to open up the eyes of the shoppers here and realize that we can all learn from sharing information. I know that many are afraid to post publicly.

I think it's good that some care enough to offer input, though.

I can see why one would wonder about the experience of those who voted, but really, how much experience does it take with a company to know that they are terrible to work for? If you started MSing anytime in 2009 and took shops for a company that stiffed you, your input would have been valuable to the survey.

In my opinion, a company that has years of positive feedback and then starts to go bad is the most dangerous. Just like the shoppers that burn out, they are capable of doing more harm because they are trusted.

That's what's problematic about disseminating data like this. It has a very short self-life.

Until there is a better system, the message boards do a good job at keeping all of us updated on who's good and bad. It just takes a lot of regular reading and I feel bad for the newbies who don't know which companies to go with.

Steve, please know that some of us really appreciated your doing this. Some of the comments are sooooo picky! I agree with the better part of the results entirely! Thanks.

Wild... some of my most "un-favorites" are at the top of the list!

Thanks for compiling the survey. I agree with the results.