<br><br><div class="gmail_quote">On Fri, Jul 27, 2012 at 11:53 AM, Florin Iucha <span dir="ltr"><<a href="mailto:florin@iucha.net" target="_blank">florin@iucha.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Fri, Jul 27, 2012 at 10:59:25AM -0500, Robert Nesius wrote:<br>
> On Fri, Jul 27, 2012 at 7:20 AM, Florin Iucha <<a href="mailto:florin@iucha.net">florin@iucha.net</a>> wrote:<br>
><br>
> > On Thu, Jul 26, 2012 at 11:44:05PM -0500, Robert Nesius wrote:<br>
> > > With iSCSI, where is the filesystem management overhead?Is the<br>
> > > filesystem overhead is on the client side with the server just receiving<br>
> > > low-level I/O operations that go straight to disk, whereas with CIFS the<br>
> > > server is having to handle mapping the I/O from the filesystem layer<br>
> > > through to hardware layer, causing it to be slower on it's responses<br>
> > > (ACKS)? I've never worked with it myself... just curious.<br>
> ><br>
> > Yes, that's how it works. However, I am measuring the performance of<br>
> > the system composed of the two machines (plus the switch) and iSCSI<br>
> > shows twice the performance of CIFS. Somebody has to do the<br>
> > filesystem dirty work, be it on the client or on the server.<br>
> ><br>
> > I could see where you have a workload that you spread across two<br>
> > sub-systems and if one of them reaches capacity, that limits the<br>
> > throughput of the entire system (some variation of Amdahl's law).<br>
> ><br>
> I think there is more post-processing on the server side<br>
<br>
The server load is 5% in both cases.<br></blockquote><div><br></div><div>Sorry Florin - I didn't explain my thoughts very clearly...</div><div><br></div><div>I meant more post processing in the form of micro reads/writes to the underlying filesystem in CIFS. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> and that's going<br>
> to slow down server responses and that's going to push back upstream and<br>
> cause writes to wait.<br>
<br>
On the contrary, with CIFS, more data gets sent through the network,<br>
because the client keeps track of cluster allocation, reads and<br>
updates the filesystem metadata. With CIFS is "here's the file,<br>
kthxby".<br></blockquote><div><br></div><div>I'm going to assume that first CIFS was meant to be an iSCSI, and if so then what you said is exactly what I was thinking. And I don't think the issue is the amount of data being sent over the network in either case. It's the number of interactions! required between the client and server, and the server and the underlying disk. Yes, the iSCSI system is tracking more than the CIFS system, but when it comes time to throw a lot of bits at the storage medium, iSCSI is optimized to get it done in a more streamlined fashion. Wasn't the whole point of iSCSI to sidestep the inherent slowdowns/inefficiencies with network filesystems like CIFS? You're exposing a block device to the network instead of a filesystem... </div>
<div><br></div><div><a href="http://jpaul.me/?p=787">http://jpaul.me/?p=787</a> is showing exactly what you're reporting, btw. </div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> > But both boxes are very powerful (3.3GHz 6-core for workstation,<br>
> > 2.8GHz 4-core for server) and completely idle that neither is the<br>
> > bottleneck. I'm writing a 11GB file, and the server has 16GB of RAM.<br>
> ><br>
> If they are idle then that suggests to me they are blocking for I/O<br>
> (waiting).<br>
<br>
Of course they are waiting for network IO. The question is why is<br>
CIFS waiting for network IO and what can I do to reduce that wait.<br>
<br>
> Are you writing to a RAM DISK?<br>
<br>
No, I'm writing to a SATA disk that can write sequentially more than<br>
100MBytes/second.<br></blockquote><div><br></div><div>Just wondering why you mentioned your server has 16GB of RAM since that's irrelevant then, unless you're pointing out there's no way it's swapping. 100MBytes/second throughput on your hard drive is also irrevelant if the disk-access patterns are sending the heads around to update underlying filesystem structures. </div>
<div><br></div><div>Try allocating a 12 GB RAM DISK on your cifs server and serving it up via CIFS and then watch CIFS close the gap to iSCSI when you drop your 11gb file on it. </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> > The CIFS results were so bad, I was concerned there was a problem with<br>
> > the hardware. It just occurred me two days ago that I could flip things<br>
> > around, use iSCSI and test the system performance in a different way.<br>
> ><br></blockquote><div><br></div><div>But as you pointed out later, that's Apples and Oranges.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> I'm thinking of ways to test my hypothesis. I'm not sure changing the size<br>
> of the TCP window on the cifs server will have an impact if the CIFS client<br>
> is blocking and waiting for a response to a write.<br>
><br>
> Maybe watch each of your tests with wireshark and watch latencies?<br>
<br>
How would that help me? What would that tell me that I don't know<br>
already? CIFS and iSCSI use different ports, different protocols,<br>
different services...<br></blockquote><div><br></div><div>It would start to illustrate differences in packet sequences for one. </div><div><a href="http://www.codefx.com/CIFS_Explained.htm#_Toc525982096">http://www.codefx.com/CIFS_Explained.htm#_Toc525982096</a></div>
<div><br></div><div>Anyway, just trying to help. Good luck with it, if you're still spending time on it. </div><div><br></div><div>-Rob</div><div> </div></div>