May 5, 2015 at 6:56 am
Hi,
I' am currently in the process of designing the Data Warehouse spec for my company
(Start up Business),
During discussions a number of the technical leads and C# developers have suggested
we use a No-SQL cluster to store the data and Apache Hive to
process into the Warehouse. It seems a bit overkill as the production system hasn't been developed yet and we have no idea of the data throughput.
Their argument is that in 3-4 years RDBMS will be obsolete and that most systems
will be running off big data clusters.
just wondering if anyone had any thoughts on this?
May 5, 2015 at 7:02 am
Sure.
Hahahhahahahahahaha!
Hehehe..
*snigger*
How you store the data and how you process it should be determined by the type of the data, the type of analysis and the type of results.
Is the data relational or is it more of a schema-less, document-type data set? That's the question that determines where it's stored, not whether someone thinks their crystal ball is better than yours.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
May 5, 2015 at 7:05 am
"not whether someone thinks their crystal ball is better than yours"
I'm thinking its more of a case of this.
May 5, 2015 at 7:07 am
I was on a training course a few months ago with Dejan Sarka as the lecturer. It was his opinion that RDMS is here to stay and there is a shift away from Data Warehouses due to the complexity, timescales and cost to implement them.
May 5, 2015 at 7:10 am
SimonH (5/5/2015)
"not whether someone thinks their crystal ball is better than yours"I'm thinking its more of a case of this.
It is (I have a couple of those types of devs here).
Ask them to formally lay out the technical advantages of their suggestion, taking into account the type of data, type of analysis, expected throughput, etc. That puts the work back onto them and turns it from "Well *I* heard that MagicSystem42 is better." into a list of pros and cons that can be examined without all the emotion and ego that usually go into these kinds of arguments.
Edit: And yes, there's a definite move away from massive, expensive datawarehouse projects, but not because relational databases are dying. Because they tend to take too long and because business requirements change too fast. There's a lot of things happening in the BI space that I'm not too familiar with (Can't spell BI personally), around alternative analysis methods and faster response to changing business requirements.
Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability
May 5, 2015 at 7:29 am
Ummmmm....
No.
NoSQL used to mean, no gol darned Ess Que Ell because we hate the DBAs and we hate the language and we just want to go FAST FAST FAST and this stuff slows us down, so NO ESS QUE ELL for me MOM!!!!
Then...
Someone in the business asked for a report. What's more, that report sliced across the data in some fashion that the NoSQL DBMS just couldn't handle. "But" spluttered the business person, "We used to be able to get exactly this kind of report when we were on a SQL database" Although they wouldn't have specified the database in that fashion really.
Hence was born, Not Only SQL, from the same ranks that were declaring the death of structured storage seven or eight years ago (still not dead, in fact, going strong and growing). The idea is, capturing data, that's one process. Reporting on data, that's another. Where applicable and where appropriate, split the two so that each one can be engineered at its best.
Now, my crystal ball. The future of BI and Analysis isn't going to be giant stacks of unstructured data. The future is with things like Redshift [/url]and Azure Data Warehouse[/url]. These are radical departures away from a traditional SQL Server installation, but, both are still relational in nature.
Just think about it. The reason relational storage has survived for so very long is because it works extremely well (when done correctly). It's efficient (when done correctly). It's flexible (when done correctly). It's very fast (when done correctly). It supports both data collection and reporting (when done correctly).
Notice the trend. The problem with relational storage, most of the time, is not the technology. The problem, most of the time, is with the implementation.
By the way, for what it's worth, I'm a huge advocate for picking the right tool. NoSQL databases such as Mongodb, Hadoop and DocumentDB, are excellent tools and should absolutely be used where appropriate. Let's just be sure they're being used appropriately.
----------------------------------------------------The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood... Theodore RooseveltThe Scary DBAAuthor of: SQL Server 2017 Query Performance Tuning, 5th Edition and SQL Server Execution Plans, 3rd EditionProduct Evangelist for Red Gate Software
May 5, 2015 at 12:45 pm
Add me to the list of people who don't think the relational model is doomed for obsolescence any time soon.
To Gail's point, a system has to be capable of being adjusted to react to business needs and it should not take 2 years to implement. Like Grant's said, the data collection and reporting are different activities.
When done right, the RDBMS performs well, but like so many technologies, it has to be done well to work well.
May 5, 2015 at 12:54 pm
Heh, I'd be willing to bet C# becomes obsolete long before relational databases 🙂
Viewing 8 posts - 1 through 7 (of 7 total)
You must be logged in to reply to this topic. Login to reply