September 18, 2009 at 4:32 am
Proposed designed architecture
We are developing a website with following details:
The basic functionality of the site is
1. The website has two portions Admin & Client.
2. Admin portion manages user and site contents and generate reports. No. of
users hitting that portion are expected to be few hundred at max.
3. Client portion is the mostly used portion and No. of users hitting that
portion are expected to be 100,000 at a time.
4. As a whole,
a. The site provides online video trainings to individuals through web
interface.
b. The web application capture tracking information like when
employee/visitor started viewing online training and when viewing was
completed. The web application automatically confers credit to individuals
based on tracking information.
c. Both visitors and admin then can view individuals' credit in the form of
reports.
The basic goals of this design are:
1. Serve 100,000 user with response time not more than 3 seconds.
2. The website down time should be closest to zero.
3. Maintenance cost for the designed architecture should be very small or
zero.
The proposed design is as follows:
Diagram #1:
Explanation of the picture "SQL Cluster with Merge & Transactional
Replication"
No. in Picture Description
1 We are targeting 100,000 no. of users for providing online video trainings
at a time
2 There would be a load balancing device that will share visitors requests
among three online web servers WEB1, WEB2 and WEB3
3 The three web servers will be running the same website (as to balance the
load)
4 Server side caching will be implemented (using MemCache logic) in web site
application. This is to minimize database hits, gain performance and hence
achieve better user experience.
5 For saving and updating users online activity/transactions in database,
web server will use Frontend SQL Cluster connection
6 For Admin site configuration and management, reports, uploading training
videos etc, web server will use Backend SQL Cluster connection. Also if the
data in reports requested by visitors would be older than a week then it
would be also served from Backend SQL Cluster
7 To distribute the load of Visitors and Admin users, we are implementing
two SQL Clusters as explained in diagram. Frontend and Backend SQL Cluster.
8 Tracking Database will keep only last 7 days data. Rest of the data will
be purged on daily bases. This DB contains users' site navigation activity.
Viewing of online trainings etc. is logged in Tracking DB. There are total
12 tables in this DB. Out of 12 there are 2 tables that would be updated by
Backend DB later on for some business needs. The change in data by Backend
Database requires updating Frontend database if the data is within a range
on past 7 days, so we need to configure Merge and transactional replication
both. See diagram 2 for more details
9 General DB contains most of the lookup tables data, i.e Users' Profile
Data, online Trainings information, Training schedules, site's
configuration, visitors locations/department hierarchy to identify users and
provide training material based on user location. There are around 45 tables
in this DB. Out of 45 tables 9 of them would be updated by Backend DB as
well for business requirement. For example. If a new user hit our website
first time and its information is not preset in User Profile table (Table
updated by ILM 2007 one time daily), we will immediately add this user in
User table. So it should be replicated to Backend Database for reporting
purpose. Most of the General Information tables would be cached on Web
Servers to minimize querying database as mentioned in Point No. 4 above.
10 Exception Log DB is self explanatory. This db is used to log users errors
while surfing the site. This db would be backed up and provided to
development team for analyzing and fixing issues faced by users
11 Backend SQL Cluster database on Node 1 will consist Tracking and General
database tables both and will be used for site management by Admin users to
update lookup tables, reporting, Upload training materials etc. as mentioned
in Point No. 6 above.
12 In Backend SQL Cluster, Node 2 would be Passive to Node 1 and incase of
any failure of Node 1, Node 2 will provide services.
13 Filtered Merge Replication would be setup for Tracking Database to keep
only 1 weeks data in Frontend. All data would be present in backend SQL
Cluster.
14 Merge replication and transactional replication would be configured for
General DB.
15 Backend db will also receive information from 2 other servers like
ILM2007 Server which provide up to date Active Directory users information
in Users Profile table. So this table would require Merge Replication setup.
16 Similarly Schedules are maintained from third party Server and require
Backend Cluster db to be updated immediately. It also update/fix users
training/transaction data upon successful updates to third party servers via
web services
Diagram #2:
September 18, 2009 at 7:26 am
Shamshad Ali (9/18/2009)1 We are targeting 100,000 no. of users for providing online video trainings
at a time
2 There would be a load balancing device that will share visitors requests
among three online web servers WEB1, WEB2 and WEB3
:pinch: Three web servers to serve 100,000 concurrent streaming video customers?
_____________________________________
Pablo (Paul) Berzukov
Author of Understanding Database Administration available at Amazon and other bookstores.
Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.September 19, 2009 at 1:27 pm
Three web servers to serve 100,000 concurrent streaming video customers?
We have CDN, setup on CISCO devices more than 130 locations and we are planning to upgrade the solution for streaming. Don't worry about the streaming hardware issue. we have another department which is working on it. Each user is served from its nearest Content Engine Router for streaming. Web servers are just used for user interface and tracking user viewing of online trainings etc and for reporting purpose.:-)
Shamshad Ali
Viewing 3 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic. Login to reply
