What is the best way to execute ~100 I/O bound scripts?

Question

What is the best way to execute ~100 I/O bound scripts?

pokalsing

Valued Member

Points: 63
More actions
April 26, 2022 at 5:32 pm

#4022619

As the title suggests I am looking into ways of executing a lot of scripts as fast as possible. This is something I’ve never done before so not entirely sure how to approach it just yet.
Originally, I thought of setting up a cloud formation template with the same number of instances as there are scripts but that just seems very inefficient.
Then I thought of setting up an instance with multiprocessing and a queue which the instance can pull the scripts from. But, then again I’d be limited to the cores available. So, I’d have to set up multiple EC2 instances and have them all pull from the same queue which still doesn’t seem like the best choice available.
Does anyone here have an idea of how to do this?
May 19, 2022 at 8:07 pm

This was removed by the editor as SPAM

Viewing 5 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic. Login to reply

Jeff Moden SSC Guru Points: 1000539 More actions · Answer 1

What do the scripts actually do? The reason why I ask is that if there's contention, you could end up with quite a bit of blocking that would make this exercise basically single threaded no matter how many core/CPUs you throw at it.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.
"Change is inevitable... change for the better is not".

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)
Intro to Tally Tables and Functions

Grant Fritchey SSC Guru Points: 397721 More actions · Answer 2

If the question is, how to run a bunch of scripts at once, I'd suggest taking a look at Powershell threading. It's a pretty easy way to launch a whole slew of processes all at the same time.

Otherwise, are you asking how to tune the queries or how to set your hardware to overcome I/O issues? As Jeff says, what are those scripts doing?

Sorry, but the question is a little unclear.

----------------------------------------------------The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood... Theodore RooseveltThe Scary DBAAuthor of: SQL Server 2017 Query Performance Tuning, 5th Edition and SQL Server Execution Plans, 3rd EditionProduct Evangelist for Red Gate Software

Jeff Moden SSC Guru Points: 1000539 More actions · Answer 3

pokalsing wrote:

I got this,..

Excellent... please share what you've got.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.
"Change is inevitable... change for the better is not".

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)
Intro to Tally Tables and Functions