IVR Capcity

thx Ian, I'm really not expert in this. Above all I'm not a technician. Some Telco's tell me that lines are not a problem. They told me about Cysco platoforms they use that can manage many more than 100k.

But anyway, I would leave this matter to telcos. Problem is after them: which platform must I have to manage that huge ammount of calls?

In thoery is possible to recive call in one country (ex: europe) and interact with them in a data center in another (ex: usa)?

Please give me also kind of site / community that can be useful :D

I'm trying to do a FEASIBILITY STUDY to understand if is possible to manage 100.000 calls in the same time:

a. users will just interact with a menu, not speaks with anybody b. they must stay on line no more than 30 min. c. they will interact in the same time d. menu depht is 7 levels. e. calls are fromm the same country

I'm not in that business: what I understood is that:

1. bandwith is not a problem. Telco's should be able to manage this number of calls using IP technology (SIP?)

??????????? I thin you will find that bandwidth will b a problem

1. Big IVR can manage more or less than 40k calls each
   formatting link
   this was based on 8 units running 5000 channels each NOT ivrs

1. medium IVR like this costs more or less 20k ?

Doubt that very much as the one you mention is based n SGI equipment

My question is: how manage 100k users interacting at the same time for

30?

First do your maths on the bandwidth and codec you want to use thenfind a data center that has that.......

Oke.

What sort of menu is this? Is is database connected, does it play wave file/mp3 or any other kind of interactivity? Should it double as ACD or is it needed for some kind of mass-media call-in (tv?) game.

Don't understand this requirement.

All 100k will also call at the same time or is there a max of 100k users that can be connected to this system?

See questions under a.

This is not important for the system itself. Possibly it is important for the bandwidth/number of lines.

I would double check. Consider that you need (depending on the codec) somewhere between 15 and 70 kbps per call and the link needs to have excellent quality in terms of packet loss, delay and jitter. In other words you need a fairly big pipe.

Also wondering where these 100k calls originate. Do they originate in the IP domain or in the PSTN. In other words do you needs gateways as well?

This numbers are not realistic. You'd need a fairly big system to connect these numbers of users. Please consider things like possible database interactivity (maybe only for configs/cdr), cpu power needed to playback audiostreams to users, all kinds of limits imposed by operating system or hardware. I'd say you need a pretty heavy system to do this.

Also you need to make some kind of deal with the telco. They need to provide bandwidth, QoS and gateways. All of these are typically not free and I wouldn't be surprised if the telco wants to see money for services...

If you also need gateways from PSTN to IP, I'd say you make a good deal if you get a system that supports 60 to 100 callers at any given time, fully configged for 20k euro.

Arnold.

Yes they may be able to have 100K users but that is a MASSIVE difference to

100K calls, Just take a moment and think about the bandwidth.

if you are using G729 the you will need 2,900,000 Kbits at least and if you have callers using G711 its 8,200,000 Kbits think about it thats nearly 5 and half thousand T1 links.

A very big one!! having built Vmails for 30000 users they could handle 240 concurant calls and where based on 4 quad xeon boxes clustered.

yes..

but again you would floo most links tring to get the bandwith you want through.

*SNICKER*

100k simultaneous calls is the entire switching capacity of a _major_ central office facility. You're talking about every phone number for

10 exchanges (NPAs) in simultaneous use.

look up an 'erlangs' table, and see how big a facility it takes to generate

100k simultaneous 'external' connections. You're expecting the telco to provide _that_ level of 'back-end' resources to your connection -- it will *NOT* be inexpensive.

You're 'requirements' translate to a dedicated C.O., with dedicated fiber circuits to probably half-a-dozen separate 'tandem' facilities.

That is (literally) millions of dollars in gear. without counting the hundreds of thousands for laying those new fiber circuits, and other 'one-time' costs. Without even touching the 'recurring monthly charge' for things.

I'd be _really_ surprised if the recurring was less than a quarter-million dollars/month. not the least surprised if it was over $700k/month. Just for the 'capacity' from the telco.

I wouldn't be surprised if it costs you $50-100k to get a reliable *estimate* of what the 'real'/'actual' costs of such service would be.

*GUFFAW*

high capacity 'line card' interfaces are multiple dollars per line. You need several dollars/line of CPU and supporting resources per 'in use'.

you don't do it for anything in the '20 cents a port' range.

Scale up by a factor of 10x-30x, or more, for 'real world'. just for the 'hardware' costs. *before* you start figuring in redundancy, and the excess capacity needed for dealing with equipment failures. (if you have a component with a MTBF of a million hours that you need for _each_ call circuit, you're going to have -- on average -- 2-3 failures of that component *every*day*. Multiply that by the number of _different_ million-hour MTBF devices affecting each call, and you begin to grasp the scope of the problem. You *have* to 'design in' automatic handling to 'route around" failed components. This drives costs up. a *lot*.

"with great difficulty".

basic 'call management' is one thing.

Doing 'whatever is needed' to support that 30-minute duration session, presumably data-base lookups, text-to-speech rendering, etc. is a whole 'nuther kettle-of-fish. Bring a big checkbook.

There are a couple of basic approaches to the 'problem' you have presented.

1) build a _bunch_ of 'medium-sized' systems -- say, ones that can handle a few hundred simultaneous calls -- and deploy enough of them "in parallel" to handle the current load-level. This 'scales' fairly linearly on the hardware side, although there are management/support issues. 2) Design a *BIG* mother of a system to handle the anticipated load. this will be a _long_ and *expensive* process. You require connectivity in the "multiple TERABIT" range. something on the order of 4,000 'gigabitEthernet' interfaces.

Assuming you can find a 20gigabit interface, you still have to support 200 of those.

*AND* you need 'backplane' capacity in the _10s_of_terabytes_ per second. yes, "bytes", not "bits". You've got to move all that 'voice data', *TWICE* (once from storage to 'RAM', and then from RAM to the 'line'), *plus* processor 'code', plus all sorts of other 'overhead' stuff.

Implicit in that is the ability to get data to/from memory at the required speeds. Call it 'nano-second access times', *with* 1000-way interleaving.

Do you begin to get the idea of the design issues?

May you please explain to me better the design issues of point 2? My only requirement is to work with a "dial tone" choice system in order to cross a level based menu, structured like a tree data structure. I don't need to use voice I need only to make a menu and track the choices. Are the two solutions proposed still the best approach?

Huh?!!!

100,000 simulations calls can be handled 25 DS4s. At 274.176mbs each, that is a total of 6854.4 mbs, which equates to 7x 1gbase-x, or a single 10gbase-x. Where are you getting your numbers from?

Seems that there may be some QOS (quality of service) issues with 10Gbase-x as its packet based and it may not be able to give you full bw in real time for what you are planning. I know that the old 10BaseT do to CSMA/CD would max out at roughly 33% of BW due to collisions.

Mixing up cell switching networks and data packet networks can be misleading. Data packet networks do not have time critical delivery hence QOS problems.

Just a thought...

Chris Hills wrote:

First off, in my original 'off the top of my head' numbers, I slipped a decimal point. Aggregate throughput needed is 'merely' in the 5-10 gigabit range, not terabits. This requires an OC192 circuit, "at least". probably more like three 0C96 circuits. With a multiplexor/de-multiplexor to interface to 10-15 separate gigabit-Ethernet 'local' network segments.

However, dealing with even that level of network traffic in a single general- purpose computer, presents a number of challenges. if, for convenience, you use 8gbit/sec, that equates to 1gBYTE/sec. To output data at that speed, you have to be able to transfer it from main memory to the output device at that speed. *AND* you have to get it into main memory from some form of 'secondary' storage. that means you have to have 'internal' data transfer speeds of _2_ gBytes/sec. Plus the requirement for moving the 'instructions' for the CPU from memory to the CPU itself. It is not uncommon for the volume of 'instructions' to be larger than the volume of 'data' being moved.

To get this kind of multi-gigabyte/second data rate, on internal data-paths within the computer, you have to have 'average access to data' speeds that are in the fractional-nanosecond range. Present-day memory technology misses that level by roughly two orders of magnitude, for an individual memory access. Thus, you have to use multi-byte 'wide' access. Modern high-end micro- processors support _eight_byte_ wide memory access. some memory-systems use cache buffering at that level, with 16- or 32-byte wide interfaces behind the cache to physical memory. For this kind of an application, caching of _data_ does *not* improve performance noticeably -- it affects only reading from memory, _not_ the writing to the 'telephone interface'.

Fast SDRAM (DDR 3000) has a "cycle" time of 5ns. with AAD being triple that (latency of 3 cycles) 8 bytes in 15 ns is only 1 byte/2 ns. And we need 1 byte / 0.2 ns. This is a problem!

'Conventional' architectures can (albeit barely) keep up with the capacity of a single gigaBIT interface. Your application is looking at gigaBYTE data rates. You don't get that in a single system, using 'commodity' hardware.

When you leave the realm of 'commodity' hardware, the cost for specialized systems goes up precipitously.

Note the above calculations do _not_ consider the problems that arise when multiple calls need access to the same data _at_the_same_time_ -- e.g. playing the same menu selection to several users, simultaneously. Dealing with _that_ issue requires 'faster' access to data; so that the same data can be accessed several different times, within the timing requirements.

"voice" bandwidth is the only way to communicate to/from an end-user via the public switched telephone network. You 'play' recorded audio tracks that describe the menu choices. You 'listen' for the audio tones that are the "touch tone" (DTMF) by which the user makes the menu selection. Even though the user is not 'speaking' to the computer system, the 'voice' channel is still required, to pass the touch-tone 'commands', and the 'voice responses'.

You have a requirement for a certain level of capacity.

There are, essentially, _only_ two basic ways to get that level of capacity.

1) a single very large system -- what is called a 'monolithic' solution. 2) multiple small systems used 'in parallel'. this is commonly referred to as the 'divide and conquer' approach.

(Note that in the above, there is _nothing_ being said about the nature of the problem. This is a fundamental concept, applicable to any kind of design engineering.)

For 'sophisticated' design situations, one may employ a mix of those approaches. Example: doing database queries -- one may have a 'monolithic' back end system that is the actual database and processing engine, and a large number of 'paralleled' front-end units that do actual user interaction, query formatting, input validation, etc, and then hand off the 'optimized' already-validated request to the back-end, and, upon receipt of the response from that back-end, do all the 'housekeeping' for displaying the information to the use in a 'pretty' manner.

Which approach is better -- in terms of performance, and or cost -- for which part(s) of the problem is extremely dependent on the _precise_ nature of the problem that one is designing a solution for. And on the exact "mission requirements" of the project. Things that do not "seem" like they would make a difference can, in fact, turn out to be _critical_ issues.

a sort of mass-media game/pool

The lenght of the connection will not be bigger than 30 minutes.

In my vision 100k calls is the same of 100k users

that was wrong: you will have 7 menu with 4 option each one.

Probably not.

How many connection the bigget IVR could manage?

That's sure ;)

I thought telcos manage this matter. Thx, Andrea

It's not ;-) Typically in these sort of system the maximum number of callers that can be served in the peak moments is of interest. The number of users would be of interest if you are looking at some sort of application where the user has to be identified (ie. banking applications).

It still would be of interest what will happen. I'm pretty sure that you will need to play audio to the callers to let them know what options to press in the system. This than needs to be processed and the next menu level will do the same..

You have a pretty specialistic question ;-) There are companies who specialize in these kind of systems. They are typically not telco's (as in PTT's / LECs) and you need to think bigger budget.

Regards, Arnold.

I've been reading all the posts and had a thought that there may be an easier approach. For the time being lets forget everything D suggested and look only at what she needs to accomplish.

1. support 100K simultaneous calls. Given the large number of calls I'm going to assume that the callers are not all serviced by the same CO.

1. The caller will be presented with up to 7 menus each allowing four choices.

2. She needs to know the results of the choices. It sounds more like a phone-based polling system, so I'll assume 1) that there is a database back end to the IVR, and 2) you need to know the compiled results and not what choices each individual made.

A distributed solution should work relatively inexpensive, fault tolerant, and would not overload you local CO.

So look at it this way.

1. Build 20 independent IVR systems each supporting 5K calls, or some number combination that works out as cost affective. And a few extra just in case. If you try to do this on general, Intel, type CPU you'll want to go with a smaller number per system. But there are many options to this.
2. Give each IVR it's own database server that compiles the results.
3. Co-locate one or two of the IVR/database systems in each CO. This will keep the CO's load lite, trunks to the CO should not be an issue, and even if an entire CO goes off-line the system will still work. The telco can easily handle routing one phone number to multiple locations based on availability (they do it all the time).
4. Establish IP connections between the distributed database servers and a central collection site. Since the data is prepackaged as results only, the bandwidth needed is small and it's not sensitive to latency. Use PtP, frame relay, any form of internet, it will all work.
5. If you need more protection from failure, use two central collection points and have the distributed databases pass the information to both.

Is it Feasible? Kind of depends on you budget, what is it?

Thanks Randall, I don't understand what do you mean as CO: do you mean POP? People will be normal people connected by different place in one country. As the last post said they should do a sort of poll in real time.

So they will have 7 question and each question must have 4 answer.

They can decide to not play any sound TO the users; the users will know the question in other ways, they will just listen to the answer tone.

I thought in this kind of solution but:

a. it depends strictly by the numbrer of POP. waht about if I must serve a big city from which expect 50% of calls? The problem it's the same.

b. as you say costs should be less than one unique data center. But what about the mantainance of 20 different data center located many km far?

P:S. I'm not... "she"!!! IN Italy Andrea it's male name. ;-)

RC ha scritto: