unforum

Dust · Greenhorn Joined: 19 Mar 2006 Posts: 7

All clients locked out?

**Posted:** Tue May 16, 2006 2:31 pm

poozle · Entrenched Joined: 15 Aug 2005 Posts: 1090

Re: All clients locked out?

**Posted:** Tue May 16, 2006 2:40 pm

jazzychad · Veteran Joined: 03 May 2006 Posts: 74

Just an idea...

I've had problems with MS IIS in the past... the free version limits the number of active connections to something really low, like 10 iirc. (This may have changed. I stopped using IIS a long time ago) Perhaps Apache would do a better job of handling large numbers of connections? Since we seem to be able to hit the server and get a response (albeit 403 or 500 or 503), it does mean it's connecting to the server. I know in apache you can manually set the number of connections to allow, which can be really high. From what I can gather from my own Ethereal traces, bandwidth isn't really the problem, it's the bottleneck at the webserver not allowing connections through. Do you know if there is currently a limit to the number of active connections at the server?

It also just occured to me that you might have to use IIS because of the asp/soap model you're using... any truth to that?

Again, just an idea. I'm sure you have thought of this before since you seem to know a lot about networked applications. Networking is my specialization, so I thought I might be able to contribute somehow.

Again, great work. We're all in this together.

**Posted:** Tue May 16, 2006 2:55 pm

chimera245 · Decorated Joined: 09 Mar 2006 Posts: 209

Some background first for everybody - to perhaps quell the debate that is taking place here and elsewhere.

The design for the assault was originally based upon the following premise of us going through about 25000 Work Units per day at peak.
This would go thru 50% of the key space in approximately 300 days - or 9 months. I posted this somewhere long ago.

This meant that the network and other infrastructure had to stand 25000 * 3 contacts with the client per day.

1 Times Validating the Client (assuming each client is run on average once per day)
1 To get each Work Unit
1 To result each Work Unit

This peak workload of 75,000 Web Service calls per day seems a lot, but in actual fact it works out to only a little under 1 per second.

Assuming a work unit time of 20 minutes per work unit (the numbers I had with the original client design) required approximately 1000 clients - a number which looked and was achievable.

All was good - this ran smoothly for a forthnight (1_0_3).

Then, I put in a client (1_1_0/1_1_1) which could do 1 work unit in an average of 5 minutes.

Now our 1000 clients are doing work 4 times as fast - with the infrastructure stretched to doing 100000 work units per
day, with 300000 Web Service calls. This load (4 per second) exceeds the capacity of the infrastructure, and a client is required to
deal with this. 1_2_0 is produced, and this brings the client contacts down to:

1 Time Validating the Client (assuming each client is run on average once per day)
1 To Get 10 Work Units
1 To result 10 Work Units

Great, out 1000 clients with do their 100,000 work units with only 30000 Web Service calls, an actual 50% reduction in load.

Except for a couple of things:

In making this client, I had made a lot of effort to make it robust and hands free. If there was no connectivity to the server, the client would just retry until
it got connectivity - without the tedious business of the client starting and re-starting it. Great in theory. BUT it means I have no way of terminating the 1_1_0/1_1_1 clients from connecting to the web server.
Sure I can separate them from the database - but I cannot stop them spamming the web server with requests.

Second issue - in the interests of making recovery from network interruptions as fast as possible - I set the retry times WAY too low (in 1_2_0 they were an average of 5 seconds).

This means if for whatever reason the web server is being bombarded, the 1_2_0 clients MAKE IT WORSE.

OK - enough information - I have paid work to do.

WHAT IS GOING TO HAPPEN
~~~~~~~~~~~~~~~~~~~~~~

1) The Web Services are down for a few days - they will stay down until clients stop bombarding the server
2) SHUT YOUR CLIENTS DOWN - WHATEVER RELEASE THEY ARE
3) I have a 1_3_0 client ready, this features the following:

a) The Fast 1_1_1 algorithm (does a work unit in 2.5 minutes)
b) Communication with the server every 20 work units (to reduce network traffic even more)
c) A 30 - 60 minute retry period
d) Notification of Upgrade facilities
e) A Server push kill switch

4) This will be released slowly once 1 and 2 are completed. You will need this new client to operate.

**Posted:** Tue May 16, 2006 10:01 pm

EvilGenius

A likely story. What's really going on is that Chimera245, if that's his real name, has actually coded his app to secretly embezzle money from all our bank and PayPal accounts, in drips and drabs, and this recent "downtime" is just his way of covering his tracks now that he has stolen for himself a tidy sum. I expect version 1_3_0 to just be a splash screen saying, "so long, suckers!" Verrrrry sneaky. But I shall have the last laugh because I have no money to steal - muhahahaha!

A nice try for a rookie. I shall dub thee "Altogether-too-helpfulGenius"

BTW - keep up the awesome work; you're my hero *swoon*

**Posted:** Tue May 16, 2006 11:11 pm

_________________
Yogurt, curd, cream, cheese and butter's made from liquid from my udders - I am cow!

QBKooky · Decorated Joined: 24 Jul 2004 Posts: 281

Thanks, chimera, for all the crazy work you've done on this! That post is very helpful for explaning to those of us not-quite-so-savvy people what the problemos with the client are.

Looking forward to 1_3... you're my hero! Jetpack

PS: Should your most recent post be double-posted to the 13labout LJ and .tk site? It took me awhile to find this post and close my 1_2_0 client.

**Posted:** Tue May 16, 2006 11:59 pm

_________________
A clue!

Misroi · Veteran Joined: 03 Apr 2006 Posts: 133

Yes, Chimera, three cheers for you and your strong code fu!

**Posted:** Wed May 17, 2006 12:15 am

chimera245 · Decorated Joined: 09 Mar 2006 Posts: 209

**Posted:** Wed May 17, 2006 1:32 am

marky1124 · Boot Joined: 23 Dec 2005 Posts: 31

Good work Chimera. I particularly like the server kill.switch push. Nice. I hope everyone shutsdown their clients asap so we can all push on with the task in hand.

**Posted:** Wed May 17, 2006 5:32 am

poozle · Entrenched Joined: 15 Aug 2005 Posts: 1090

Maybe people will get the picture when the client just refuses to connect for a few days.

You're doing a great job on this Chimera, maybe you could seen an email to the sentinel to let people know whats going on with that (or just to mind candy asking them to put it in the next newsletter). We need as many people as we can to see this message so whatever we can do to get the coverage higher would be very useful.

**Posted:** Wed May 17, 2006 5:56 am

arnezami · Veteran Joined: 14 Apr 2006 Posts: 136

Thanks for all the work Chimera. You're doing great work Wink

.

The 1.3 client seems to solve most (if not all) problems. Looks very good.

My only advise: I think it would be wise to let the 1.3 version use a different port. This would give you the possibility of rejecting connections from older clients on a very low (hardware) level.

Just my 2 cents Smile