August 22, 2010 at 6:50 pm
Exactly what I tought... completly out of my league network wise :hehe:.
Seriously, that's what I call expert advice.
We do have 3500 clients and we had 100% response rate for people wanting this feature. Now I highly doubt that they'll all signup for this once they know what they have to do to get in with the program (there's a lot a work to map all the sku # from our erp to theirs, among other things, and of course all the setup for secure data access, which may include extra hardware and certainly a good amount of people ware). Also I've begged for a smaller trial with our best clients for the first "release".
The main thing for me is that I don't have to coordinate all that data entry and network setup, I just have to show them what to do and they take care of it from there on out. It's also their call to add as many clients as they want... or can handle to.
Last question. We currently only have T1 access for our entire building and extranet (which is not enough at the moment). We've ordered a T2 but now I'm wondering if it can handle all that traffic?!?! Correct me if I'm wrong but using encrypted / vpn connections must certainly increase the bandwith usage for that data transfer?!
My understanding is that if you encrypt at 128 bit it'll take 128 times more bandwith to communicate your message back and forth. So assumming 2 searches per second avg and 1K of real data exchange, we're talking about 256 KB/S of bandwith.... almost 20% of the total new bandwith. Are those calculations valid?
Thanks again. 🙂
August 23, 2010 at 2:12 am
No, that's definitely not right. Encryption will make the data larger, but that's only because encryption algorithms usually have a fixed block size to deal with. For 128-bit encryption that block will be 16 bytes, so whatever you send will be scaled up to the nearest multiple of 16 bytes--e.g. to get a 128x inflation of size you'd have to be sending a single bit of data, which doesn't seem very likely! I wouldn't expect an overhead of more than a couple of percent in most cases, and I don't think anyone would ever use a VPN if that wasn't the case.
August 23, 2010 at 4:23 am
It bothers me that data latency is an issue in this situation. The main problem I see is network hubs going down or your clients having modem lines instead of fiber. You're going to have data latency issues, whether you build them into your system or not. It takes time for data to move from client site to your site and back to client site. There is no 1 second magic bullet, no matter how hard you look for it. There's gonna be lag.
You can't promise instantaneous responses because you don't control all the environments and the data is spread out so far between clients. And if you try to promise it, and don't deliver, that's going to make your company take a serious reputation hit.
Talk to your bosses. Make it clear that no matter what solution is implemented, that data latency of some sort will be a real, and uncontrollable, issue.
August 23, 2010 at 4:32 am
paul.knibbs (8/23/2010)
No, that's definitely not right. Encryption will make the data larger, but that's only because encryption algorithms usually have a fixed block size to deal with. For 128-bit encryption that block will be 16 bytes, so whatever you send will be scaled up to the nearest multiple of 16 bytes--e.g. to get a 128x inflation of size you'd have to be sending a single bit of data, which doesn't seem very likely! I wouldn't expect an overhead of more than a couple of percent in most cases, and I don't think anyone would ever use a VPN if that wasn't the case.
Thanks.
August 23, 2010 at 4:40 am
Brandie Tarvin (8/23/2010)
It bothers me that data latency is an issue in this situation. The main problem I see is network hubs going down or your clients having modem lines instead of fiber. You're going to have data latency issues, whether you build them into your system or not. It takes time for data to move from client site to your site and back to client site. There is no 1 second magic bullet, no matter how hard you look for it. There's gonna be lag.You can't promise instantaneous responses because you don't control all the environments and the data is spread out so far between clients. And if you try to promise it, and don't deliver, that's going to make your company take a serious reputation hit.
Talk to your bosses. Make it clear that no matter what solution is implemented, that data latency of some sort will be a real, and uncontrollable, issue.
Ok, awesome warning. I was aware of the fact that I can't garantee anything and my bosses are aware of this, but I do need to maximize the chances of a fast connection.
Also our situation might be a little unique. I'd say that 99% of our clients are located within 400 miles of the main office. And 100% are in Canada (Québec, Ontario).
So did you experience that latency problem when doing something similar worldwide, or even locally (assuming each client has T1 access in his building or fast cable connection)?
Also keep in mind that for each of the separate clients, we might exchange 5-20 MB of data (after encrytion) over an 8 hours day. With each transfer being at most 20 KB total (extreme case). So even on a slow connection, 20K shouldn't be all that big of an issue (I'd think)!
I'm ok with 1 sec to connect even if I'd prefer 0, but I don't want it to to sporadically jump to 5-10 secs while the clerck waits on his data.
August 23, 2010 at 4:53 am
Also our situation might be a little unique. I'd say that 99% of our clients are located within 400 miles of the main office. And 100% are in Canada (Québec, Ontario).
So did you experience that latency problem when doing something similar worldwide, or even locally (assuming each client has T1 access in his building or fast cable connection)?
We have a setup whereby all the branch offices are connected to Head Office via VPNs--the furthest one is 40 miles away, so this is hardly a wide-spread setup. We sometimes have major latency issues--something as simple as someone at a branch sending a large e-mail can cause the upload bandwidth there to be flooded, and suddenly you're looking at 400+ ms ping times and major latency on the connection. Since you have no control over what your customers are doing with their network connections you might well have the same issue.
August 23, 2010 at 4:53 am
paul.knibbs (8/23/2010)
No, that's definitely not right. Encryption will make the data larger, but that's only because encryption algorithms usually have a fixed block size to deal with. For 128-bit encryption that block will be 16 bytes, so whatever you send will be scaled up to the nearest multiple of 16 bytes--e.g. to get a 128x inflation of size you'd have to be sending a single bit of data, which doesn't seem very likely! I wouldn't expect an overhead of more than a couple of percent in most cases, and I don't think anyone would ever use a VPN if that wasn't the case.
A couple of percent is near enough, may even be slightly higher than the real overhead here (assuming the query is shorter than the result set, we can ignore the query since a T1 line is symmetric in bandwidth - and if customers have something asymmetric there low capacity will be upload of the result set to the central server, so the overhead on sql text transmission is irrelevant) given what we know about this app. And that overhead is lost in the variation of bandwidth practically available caused by congestion in hubs/routers/relays and/or on links out there in the internet.
You certainly can't get anywhere near a factor of 128. as encryption overhead - the overheads imposed by TCP and IP and DIX (or whatever level 2 protocols are used) are several times bigger than the overhead imposed by encryption.
Since we are talking internet link protocols here the IP layer forces packets to be a multiple of a byte so already for that reason alone a factor of 128 is impossible, the worst would be a factor of 16, but of course that too can't happen. Since the data is held in TCP frames which are a minimum of 20 bytes plus the data, even if you are sending only 1 byte at a time (ie your network software is completely broken) the worst case overhead has to be less than about 76%. Those TCP frames are carried over IP, which has a minimum header length of 20 bytes (for IPv4; more for IPv6) so in fact the worst possible case (single byte transmissions) is less than 37%. Then you have link framing, which will vary from hop to hop, but will usually be DIX so there's another 18 bytes always there and your worst case overhead is going to about 25% - so far from getting an worst case overhead of a factor of 128 you would be really struggling to build a system with a factor of 1.25.
Of course in practise you won't be transmitting 1 data byte at a time. You'll transmit quite a bit (neither the query nor the result set will be very short) so I guess the average data length could be more than 512 bytes - that means that 512 data bytes plus 58 comms bytes get an overhead of on average 7 and a half bytes, which is less than one and a third per cent overhead.
Tom
August 23, 2010 at 5:12 am
Ninja's_RGR'us (8/23/2010)
Brandie Tarvin (8/23/2010)
You're going to have data latency issues, whether you build them into your system or not. It takes time for data to move from client site to your site and back to client site. There is no 1 second magic bullet, no matter how hard you look for it. There's gonna be lag...Talk to your bosses. Make it clear that no matter what solution is implemented, that data latency of some sort will be a real, and uncontrollable, issue.
Ok, awesome warning. I was aware of the fact that I can't garantee anything and my bosses are aware of this, but I do need to maximize the chances of a fast connection.
...
So did you experience that latency problem when doing something similar worldwide, or even locally (assuming each client has T1 access in his building or fast cable connection)?
I haven't done this sort of setup myself, but I have, in the past, been a "client" for a non-business related setup. Lag was a serious issue and most of it was caused by local network routing issues, hubs going down, lots of bandwidth traffic.
Ninja's_RGR'us (8/23/2010)
Also keep in mind that for each of the separate clients, we might exchange 5-20 MB of data (after encrytion) over an 8 hours day. With each transfer being at most 20 KB total (extreme case). So even on a slow connection, 20K shouldn't be all that big of an issue (I'd think)!I'm ok with 1 sec to connect even if I'd prefer 0, but I don't want it to to sporadically jump to 5-10 secs while the clerck waits on his data.
My warning isn't about the amount of data you're sharing and transferring. It's about the vehicle used to transfer that information. Even small amounts, I promise you, can lag badly if your ISP, your client's ISP, or the ISP in the middle, starts having issues.
Have you ever done a tracert command on local websites? I did one, once. The site I searched was a Jacksonville site and I happen to live in Jacksonville. The packets from the tracert went from my PC to the local ATT server to New York City, Boston, and then back down to Jacksonville. It took more than 20 seconds and less than 60 for the full route to be completed (I don't remember the exact number).
Test it out. Use a command prompt to do a tracert on a few of your client IP addresses and see what routes the packets take to get from you to them. I'd be surprised if most of them followed a direct route.
August 23, 2010 at 5:20 am
Ninja's_RGR'us (8/23/2010)
Also keep in mind that for each of the separate clients, we might exchange 5-20 MB of data (after encrytion) over an 8 hours day. With each transfer being at most 20 KB total (extreme case). So even on a slow connection, 20K shouldn't be all that big of an issue (I'd think)!I'm ok with 1 sec to connect even if I'd prefer 0, but I don't want it to to sporadically jump to 5-10 secs while the clerck waits on his data.
I think you definitely have a bandwidth problem. 3500 customers each delivering 5MBytes of data over 8 hours is nearly 6 Mbits/sec, your T1 line is only 1.554 Mbits/sec so it won't fit.
Even an upgrade to T2 will leave a big problem: you will be using over 77% of the available down bandwidth even if every client only sends the minimum of your range (5Mbytes); with a size range of 5 to 20 MB you are probably well over 100% of T2 capacity. (And that's assuming that your provider actually delivers genuine T2, instead of delivering T1c but calling it T2.
Tom
August 23, 2010 at 5:23 am
Not to mention, of course, that this 5-20Mb per customer may well not be spread evenly over the whole day--if you get a whole bunch of them trying to upload at the same time then you'll run into additional lag issues as all that data tries to flood down your pipe.
August 23, 2010 at 5:30 am
Add all the information Paul and Tom just gave you to the fact that you do not have a direct, dedicated link to your customers, and suddenly you have a system that will respond slowly at the best of times.
The way around this, of course, is to establish multiple direct T1 or T2 links, one per customer, that run directly from you to them, with nothing in between. But for 400 miles distance and 3500 clients? That's a seriously prohibitive cost.
August 23, 2010 at 6:10 am
paul.knibbs (8/23/2010)
Not to mention, of course, that this 5-20Mb per customer may well not be spread evenly over the whole day--if you get a whole bunch of them trying to upload at the same time then you'll run into additional lag issues as all that data tries to flood down your pipe.
Definitely the case, I'd wager that 60% of all the searches will be from 8 am to noon (assuming the trend continues).
August 23, 2010 at 6:13 am
Tom.Thomson (8/23/2010)
Ninja's_RGR'us (8/23/2010)
Also keep in mind that for each of the separate clients, we might exchange 5-20 MB of data (after encrytion) over an 8 hours day. With each transfer being at most 20 KB total (extreme case). So even on a slow connection, 20K shouldn't be all that big of an issue (I'd think)!I'm ok with 1 sec to connect even if I'd prefer 0, but I don't want it to to sporadically jump to 5-10 secs while the clerck waits on his data.
I think you definitely have a bandwidth problem. 3500 customers each delivering 5MBytes of data over 8 hours is nearly 6 Mbits/sec, your T1 line is only 1.554 Mbits/sec so it won't fit.
Even an upgrade to T2 will leave a big problem: you will be using over 77% of the available down bandwidth even if every client only sends the minimum of your range (5Mbytes); with a size range of 5 to 20 MB you are probably well over 100% of T2 capacity. (And that's assuming that your provider actually delivers genuine T2, instead of delivering T1c but calling it T2.
Thanks, I'll collect figures and warn the powers that be that we'll need more than a T2 (assuming real 10 mpbs) to handle all clients. Right now all my data is speculative so it's hard to really pin down what we need. Also we've had a ton of problems with our current isp (only one available on our street), so I'll consider that in our setup.
Moreoever, as I said earlier I highly doubt that we'll get all 3500 clients aboard... ever. So that's also another big question mark after "how fast we can get them in with our staff)... I'll have to keep a close eye on this one.
August 23, 2010 at 6:14 am
Brandie Tarvin (8/23/2010)
Add all the information Paul and Tom just gave you to the fact that you do not have a direct, dedicated link to your customers, and suddenly you have a system that will respond slowly at the best of times.The way around this, of course, is to establish multiple direct T1 or T2 links, one per customer, that run directly from you to them, with nothing in between. But for 400 miles distance and 3500 clients? That's a seriously prohibitive cost.
Even with a good rebate off current cost and only 1500 clients setup, that's 750 000$ / month, excluding setup :w00t:.
August 23, 2010 at 6:50 am
Brandie Tarvin (8/23/2010)
Add all the information Paul and Tom just gave you to the fact that you do not have a direct, dedicated link to your customers, and suddenly you have a system that will respond slowly at the best of times.The way around this, of course, is to establish multiple direct T1 or T2 links, one per customer, that run directly from you to them, with nothing in between. But for 400 miles distance and 3500 clients? That's a seriously prohibitive cost.
Not only would the cost be prohibitive, the timescale for getting the telecomms companies to provide it would probably be several years. And with 400 miles narrow-beam radio link is not going towork because you won't get line of sight, so that's not an option either (even if it made sense to consider it for 3500 clients, and anyway I don't know if anyone but an authorised telecom would be allowed to do that in Canada).
But I don't think that a direct T1 connection to each client is needed - that would be vast overkill.
I think doing the data volume calculations very carefully would be a good idea - here we are talking about 5 to 20 MB per day per client, but each interaction has a max of 20kb, so that's 250 to 1000 uses of this new feature each day for each client, which looks a bit high.
Brandie's suggestion of using tracert to discover the hop count and typical latency to each client is also an excelent idea. In fact you can't really decide how to get the network to work until you have done that.
Probably ensuring that other stuff (downloading DVD images of evaluation releases from MS; receiving emails with 20MB of attachments; and all that sort of thing) is done using internet access links separate from the ones used for this app is something that should be looked at.
If the clients are on several different ISPs give the central server a connection to each of those ISPs, or at least to any where hop counts or latencies from your current ISP look high (you have to have multiple ISP accounts - so what? it's part of the cost of doing this sort of network) because that is your best chance to reduce hop counts and latencies (it will make your subnetting more complex, perhaps, but that's not a real problem). Use a sensible mix of connection types: a small number of clients at a particular ISP might mean an ADSL connection to that ISP would make more sense that a T1 - and it's cheaper than a T1, and it probably provides more downwards bandwidth than a T2. If the hop counts and latencies discovered by testing aren't out of order (I imagine some of them will be, but by no means all) and acurate bandwidth estimates give a lower upper bound you can maybe use a couple of T2s and stick to a single ISP. But without measurements of how the internet behaves you can't know what is right, so follow Brandies suggestion before deciding what connections you want.
(Wherever I wrote bandwidth above I meant data transmission rate; I gave up using bandwidth correctly about fourty years ago when I had to communicate with non data-comms people about data-comms.)
Don't forget queuing delays in the network when trying to work out the bandwidth needed. They are going to contribute to latency, and to the variance of latency.
The easiest queuing delay to estimate is the queue at the far end of your access link for traffic to get from the internet onto that single link. Suppose you have T1 link providing the internet connection the use to reach a few hundred clients and its average load over your 8 hour day is 90%. Now a 6kbyte message has about 32 ms transmission time on that T1 link - down in the noise compared to your 1 second limit, you might think. But if you make some fairly ordinary assumptions about the distributions of message arrival times and message lengths you can calculate that the average time contributed to the latency by that 32 ms transmission is about 320 ms, almost a third of the allowed 1 sec response, and that's the average not a maximum (if my mental arithmentic is working, 58% of messages take up to 320ms, 42% take longer; about 16% take longer than 640ms and about 6% longer that 960ms, and that's just the delay from getting to the device at the far end of your T1 wire to popping out of the wire into your router/firewall, no other latency included; but the days when I could reliably do queuing theory in my head are long gone). A good rule of thumb is ensure that neither your uplink nor your downlink is utilised more than about 67% during your peak period (two thirds utilisation is where average queuing time is the same as average transmission time for poisson arrivals and exponentially distributed service times, which tends to be a good approximation for many situations).
And don't ever forget that a light plane crashing into someone's radio tower or a construction project digging through a telecom company's backbone can change the picture drastically and you have to make sure that management and customers recognise that these things do happen (rarely, fortunately).
Tom
Viewing 15 posts - 16 through 30 (of 41 total)
You must be logged in to reply to this topic. Login to reply