Take a look at this page by Christiaan Lustig. It argues for design based on "top tasks" and not particularly on the patform -- mobile vs. desktop vs. tablet. I agree with his conclusions that you should look for your users' top tasks and that these will change based on their location/context rather than on the particular platform they are using.
Sure, I still think you should start with the smallest platform -- i.e. mobile -- to force the team to consider the most basic elements of your responsive design and what is really important to your users, but he raises good points about the "locational" and "situational" elements involved in determining what is important to your customer when they visit your site in different situations.
It's a good read.