Insert 1,000 records in one second

Question

Insert 1,000 records in one second

Viewing 14 posts - 16 through 28 (of 28 total)

You must be logged in to reply to this topic. Login to reply

Ninja's_RGR'us SSC Guru Points: 294069 More actions · Answer 1

PaulB-TheOneAndOnly (6/28/2010)
Ninja's_RGR'us (6/28/2010)I was in Pass when the Lead DBA of MySpace was there to host conferences. IIRC they were handling something like 1 M queries a minute.
Exactly. Volume is what separates boys from men.
Most people don't grasp the concept that the negative impact of bad design and bad coding increases exponentially with volume. Large environments are the field where true colors are shown.

Yup back then it was taking me 3-4 months to handle that many queries (and I taught it was A LOT :w00t::w00t::w00t:). That's why I was flabergasted when I heard those numbers.

Eugene Elutin SSC Guru Points: 59322 More actions · Answer 2

Ninja's_RGR'us (6/28/2010)
...
That works... only 1 problem here, you need to have a single shared "memory" on the web server for all the executions. Then you need to batch process that (where you still need to handle reads writes and what's processed, or processing). This makes for a very dangerous bottleneck for the application...

So, basically, you don't want to buy-in? :w00t:

It's up to you! Regardless of mentioned issues above, I would still invest into the company whoes web page is going to have almost 100 million visitors per day, even if they will not be able to log all of them :-D:-D:-D

_____________________________________________
"The only true wisdom is in knowing you know nothing"
"O skol'ko nam otkrytiy chudnyh prevnosit microsofta duh!":-D
(So many miracle inventions provided by MS to us...)

How to post your question to get the best and quick help[/url]

Ninja's_RGR'us SSC Guru Points: 294069 More actions · Answer 3

Visitors are only 1 part of the equation. You still have to sell all the ads, which is harder than it seems.

The facebook guy was deemed the "youngest billionnaire". But the fact is that the site was only selling 100+ M a year in ads out of the possible billion. Yes I would take that any day, but still there's more to it than meets the eye!!

Michael Valentine Jones SSC Guru Points: 64818 More actions · Answer 4

If your IO needs are really high, especially for read IO:

MySpace replaces all server hard disks with flash drives

http://www.computerworld.com/s/article/9139280/MySpace_replaces_all_server_hard_disks_with_flash_drives

MySpace Replaces Storage with Solid-State Drive Technology in 150 Standard Load Servers

http://www.networkcomputing.com/data-center/myspace-finds-novel-use-of-ssd-technology.php

MySpace Finds Unique Use For SSD Technology

http://www.networkcomputing.com/data-center/myspace-finds-novel-use-of-ssd-technology.php

Eugene Elutin SSC Guru Points: 59322 More actions · Answer 5

Ninja's_RGR'us (6/28/2010)
Visitors are only 1 part of the equation. You still have to sell all the ads, which is harder than it seems.
The facebook guy was deemed the "youngest billionnaire". But the fact is that the site was only selling 100+ M a year in ads out of the possible billion. Yes I would take that any day, but still there's more to it than meets the eye!!

I really liked this one "... only selling 100+ M a year ...".

OK, you convinced me. I will invest into BP instead...:w00t:

_____________________________________________
"The only true wisdom is in knowing you know nothing"
"O skol'ko nam otkrytiy chudnyh prevnosit microsofta duh!":-D
(So many miracle inventions provided by MS to us...)

How to post your question to get the best and quick help[/url]

Ninja's_RGR'us SSC Guru Points: 294069 More actions · Answer 6

Eugene Elutin (6/28/2010)
Ninja's_RGR'us (6/28/2010)
Visitors are only 1 part of the equation. You still have to sell all the ads, which is harder than it seems.
The facebook guy was deemed the "youngest billionnaire". But the fact is that the site was only selling 100+ M a year in ads out of the possible billion. Yes I would take that any day, but still there's more to it than meets the eye!!
I really liked this one "... only selling 100+ M a year ...".
OK, you convinced me. I will invest into BP instead...:w00t:

I've been working for bigger companies lately (1-2 M / week sales avg). Even tho those sales appear big, they are almost equalled by the expanses.

One of them was negotiating with the union for a new contract and the union was "complaining" about all the 5-6 millions the cie had saved for rainy days. Then the depression came and the cie went from profitable to losing about 2 millions a month (even after cutting 80% of the jobs).

That's when I realized that 1 M is not that much and like sql server IT DEPENDS :-D.

LutzM SSC Guru Points: 107049 More actions · Answer 7

Hmmm, I'm curious what the OP think about the responses...

Lutz
A pessimist is an optimist with experience.

How to get fast answers to your question[/url]
How to post performance related questions[/url]
Links for Tally Table [/url] , Cross Tabs [/url] and Dynamic Cross Tabs [/url], Delimited Split Function[/url]

irmorteza2000 SSC Veteran Points: 261 More actions · Answer 8

Thanks every one for replying

We are trying to create a website that its customer is another websites. A website joins to our site and we provide some advantages for its users. So when a site with 100,000 page view joins us we must handle its 100,000 page view too. And when 10000 sites with 100,000 page view join us; we must handle 1,000,000,000 pages view and so on. So as you see there is no limit for page view.

Best regards. Morteza

irmorteza2000 SSC Veteran Points: 261 More actions · Answer 9

thanks “PaulB-TheOneAndOnly” for your information

Best Regard. Morteza 🙂

PaulB-TheOneAndOnly SSC Guru Points: 66772 More actions · Answer 10

irmorteza2000 (6/28/2010)
thanks “PaulB-TheOneAndOnly” for your information
Best Regard. Morteza 🙂

Glad to help 😉

_____________________________________
Pablo (Paul) Berzukov

Author of Understanding Database Administration available at Amazon and other bookstores.

Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.

mish_b20 Grasshopper Points: 18 More actions · Answer 11

Hello,

I believe that this question is not for this forum too - and that the solution is not very simple - because there is no simple and quick solution for that question.

Every solution has two sites of a corner.

I would suggest making a use of MSMQ or a-like solution.

I believe that you don't need all the data in a real time logged into the db - you can make use of MSMQ to that job, and to make some job in SQL + CLR functionality to insert 1000 records every 1 or two seconds .

Hardware is very important, and my colleges here are right. You should consult with someone face to face - someone who can make an estimation of a hardware you need to buy for that purpose.

Best of luck

And hope it helped.

Michael

Tony++ SSCarpal Tunnel Points: 4421 More actions · Answer 12

If you're logging activity from web sites, there's options besides logging directly. Unless there's a need for true real-time logging, look around at the other parts of your computing install and see what you can leverage.

* Have the web server(s) write their activity logs to disk, then import those logs into the database. There's also some good mining tools for IIS logs, assuming that you're using IIS. I can't tell you anything on other web servers, haven't used them.

* We're using Coradiant to monitor our web environment, and that keeps request logs that we can import into the database.

Advantages:

* the software components are already doing the logging, you're just using what you already have.

* the web servers have less work to do if they're spared the inserting

* it's all back-end, and doesn't require a SQL instance licensed for the web (per CPU or web edition). If it's collecting just for stats & analytics, you can import into a separate server(s) and take that load off the web-facing db server.

* is only as real-time as your import frequency, so lower performance may be acceptable (less hardware needed).

* you can get cookie & session information from these logs, so you have more than just the request & IP.

* and through Coradiant, we also get the bonus of the users lat/long for geographic intelligence.

TheSQLGuru SSC Guru Points: 134017 More actions · Answer 13

Usually "I" am the one (or first one) to recommend hiring a professional to help with scenarios like this. But since several others have already done that I feel ok in saying you can hire "ME" to help with this, if it is a real-world engagement. 🙂

Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service

SwePeso SSC-Dedicated Points: 39757 More actions · Answer 14

Same question asked over here last week http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=146582

I guess OP gave up.

N 56°04'39.16"
E 12°55'05.25"