The starter site that Microsoft provides with the Commerce Server product is a pretty good model to follow, so the basic site functionality was not extremely difficult to produce. However, performance tuning was a very long and painful process.
To begin with we used normal Commerce Server objects to populate an ASP.NET menu control, without any caching at all. Predictably this approach brought the server to its knees as soon as even a light load was placed on it.
Next we tried using the Commerce Server caching objects. When a user visits the page the code would check to see if there was an object for the menu in the chache, and if not then it would hit the Commerce Server objects to pull the information and place it in the cache.
This method looked great at first, and we thought we had the problem solved. The performance was fantastic. But then we started noticing that the menu was dissapearing! It would be fine for a while but would start acting up when the load increased. Performing an IISreset would solve the problem, at least temporarily.
We tried a number of code driven approaches to solve the problem, including adding checks to examine the cache to make sure there were values there. Nothing seemed to work.
Finally someone had a bright idea after reviewing the code again. The cache code we had in place was reading the data, then creating the menu, and then storing the whole menu object in the cache. Instead, we started chaching only the raw data and then building the menu fresh on the page.
Performance was not quite as good as the previous solution, but has almost completely solved the dissapearing menu problem. I have seen the menu dissapear a couple of times since then, but it is very rare.
The only thing I can guess is that someone the cache was getting corrupted and not acting like it should after that. Our servers are in a web farm, so perhaps something gets borked when a user gets switched from one server to the next in the middle of a shopping session.