Look at your resource monitor when watching a movie - I bet its at 100%. explains the buffering problem. very few pieces of software are set up to take advantage of gpu hardware encoding, so running a transcoding server requires a fairly powerful processor.
I tried PS3MS and didnt like it at all, but I wasn't using a PS3 on the other side. I use Plex for this now and like it much much better. You're going to be stuck using and modifying DLNA profiles on a PS3 regardless of which server software you use, so I don't know if Plex will be better (but I suspect it woudl be).
The problem you're [likely] having is that PS3MS is trying to transcode your files. What are the media details of it?
-container (just mkv files?)
My suspicion (and I'm not a Sony expert) is that the PS3 can't play MKV files. It probably wants .MP4 files with H264 video and AC3 or AAC audio. So PS3MS is trying to convert the MKVs into something compatible on real time, and as a result your CPU can't keep up and you get stuttering and buffering.
just for kicks, download a small free program called tsmuxer. create a .m2ts file out of your .mkv file. im 95% sure that PS3 supports m2ts. I'm curious if that will play.
A few possible solutions here:
-if the audio and video in your files is compatible, but not the container, you can try remuxing on the fly. Plex can do this, but I'm not sure how to do it with DLNA profiles. This won't convert audio or video, it will just repackage them into a compatible file type which is way less resource intensive.
-transcode to a lower bitrate. this could be the solution to a few potential problems. are you doing this over wifi? wifi sucks badly for HD video. even if your files are compatible bluray quality files cannot be reliably streamed over wifi.
if you transcoded them on the fly to a lower bitrate then you will have less bandwidth going on. that will also help your pc speed situation, because the computer can transcode to a 3000kbps 720p stream much easier than an 8000kbps 1080p one.
if you are interested in a low cost htpc look into the new Intel Bay Trail NUC. You could load OpenELEC onto it instead of Windows as a free alternative and have the entire thing up and running for well under $200