April 26, 2022 at 5:32 pm
As the title suggests I am looking into ways of executing a lot of scripts as fast as possible. This is something I’ve never done before so not entirely sure how to approach it just yet.
Originally, I thought of setting up a cloud formation template with the same number of instances as there are scripts but that just seems very inefficient.
Then I thought of setting up an instance with multiprocessing and a queue which the instance can pull the scripts from. But, then again I’d be limited to the cores available. So, I’d have to set up multiple EC2 instances and have them all pull from the same queue which still doesn’t seem like the best choice available.
Does anyone here have an idea of how to do this?
April 26, 2022 at 6:28 pm
What do the scripts actually do? The reason why I ask is that if there's contention, you could end up with quite a bit of blocking that would make this exercise basically single threaded no matter how many core/CPUs you throw at it.
--Jeff Moden
April 27, 2022 at 1:05 pm
If the question is, how to run a bunch of scripts at once, I'd suggest taking a look at Powershell threading. It's a pretty easy way to launch a whole slew of processes all at the same time.
Otherwise, are you asking how to tune the queries or how to set your hardware to overcome I/O issues? As Jeff says, what are those scripts doing?
Sorry, but the question is a little unclear.
----------------------------------------------------The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood... Theodore RooseveltThe Scary DBAAuthor of: SQL Server 2017 Query Performance Tuning, 5th Edition and SQL Server Execution Plans, 3rd EditionProduct Evangelist for Red Gate Software
May 19, 2022 at 8:07 pm
This was removed by the editor as SPAM
May 20, 2022 at 3:15 am
I got this,..
Excellent... please share what you've got.
--Jeff Moden
Viewing 5 posts - 1 through 4 (of 4 total)
You must be logged in to reply to this topic. Login to reply