May 5, 2015 at 9:12 am
I don't disagree, however in a number of companies, the DBAs provide container support and have no control over the actual implementation of the application. In that case we have to rely upon our own auditing mechanisms to capture everything that is required. Whilst one would hope that a responsible developer would include accurate error handling and app logging, many times in the past I've seen occasions where that isn't the case.
May 6, 2015 at 6:34 am
Steve Jones - SSC Editor (5/5/2015)
I'd argue auditing needs to be in the database when it affects data or legal requirements. App logging certainly should always be included.
Yes. If it is considered non-essential then possible it isn't logging or instrumentation but diagnostics.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
May 6, 2015 at 8:34 am
Steve Jones - SSC Editor (5/5/2015)
Eric M Russell (5/5/2015)
In most scenarios the auditing operation should be done asynchonously, if possible, for performance reasons. But you can't do that if the auditing operation is bundled within batch transaction.Another reason for separating the auditing operation from the batch transaction is that in the event the batch transaction fails, so does the audit event. I'm sure we're all familiar with the scenario where an error occurs, the error is logged to an audit table... but then the audit table insert is rolled back along with the rest of the batch transaction thus leaving no evidence that the error occurred.
This this isn't auditing, nor is it architected well. If I wanted to teach someone about auditing, this would be the issue. Need a variable that can be logged at the end storing the actions.
I guess I don't quite see the difference between using a variable that later gets stored from the requirement you're surprised by. It's not logged synchronously, and can fail just as easily. If the server were to lose power 2 hours into a job and the auditing is being logged to a variable - guess what you don't have when you fire up the server again....
This is the kind of requirement where service broker might really shine, since it does take the auditing of the actions out of the main process AND has guaranteed delivery. You don't want every process you have logging to a central audit table and all be held up waiting for some other process to finish the recording of whatever audit is required. Just because it doesn't happen synchronously doesn't have to mean you can't guarantee that it happens.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
May 10, 2015 at 12:18 pm
I think this one can be chalked up to immaturity. Business people would cheerfully descope a whole raft of IT concerns if they could. Auditing is not seen as adding business value, causes IT headaches and is used so infrequently that its value is not perceived.
In the UK the old FSA was disbanded following the crash of 2008 and was replaced by two separate bodies. I think I can compare the FSA to a toothless, over fed old Labrador. The FCA on the otherhand are hungry pit bulls spoiling for a fight. I suspect that auditing requirements are going to come to the fore over the next couple of years.
Apache Cassandra is great for being a non-failing logging mechanism in a polyglot system.
May 10, 2015 at 3:06 pm
I think a lot of people get "auditing" mixed up with "logging". In the former, it must not fail because there are legal reasons for it. In the latter, a lot of people capture data for the sole purpose of measuring the performance of something like how many "hits" their website took in any given time frame. The latter would also imply that asynchronous accumulation of such data might be ok because it's not critical. For the former, I want stuff to stop working if it's not being audited because of the legalities of things continuing to work that aren't being audited.
One other part that I think people don't understand as a form of "auditing" are things like "orders". If your invoice detail table can't be accessed as a form of audit for the orders taken, you need to stop the applications. Not all "audit" data appears in tables with the word "audit" in the name. 😉
Heh... as with all else, "It Depends". 😀
--Jeff Moden
May 11, 2015 at 4:50 am
Jeff Moden (5/10/2015)
I think a lot of people get "auditing" mixed up with "logging". In the former, it must not fail because there are legal reasons for it. In the latter, a lot of people capture data for the sole purpose of measuring the performance of something like how many "hits" their website took in any given time frame. The latter would also imply that asynchronous accumulation of such data might be ok because it's not critical. For the former, I want stuff to stop working if it's not being audited because of the legalities of things continuing to work that aren't being audited.One other part that I think people don't understand as a form of "auditing" are things like "orders". If your invoice detail table can't be accessed as a form of audit for the orders taken, you need to stop the applications. Not all "audit" data appears in tables with the word "audit" in the name. 😉
Heh... as with all else, "It Depends". 😀
That was what I was trying to get across but failed to do so :-S
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
May 11, 2015 at 6:52 am
Gary Varga (5/11/2015)
Jeff Moden (5/10/2015)
I think a lot of people get "auditing" mixed up with "logging". In the former, it must not fail because there are legal reasons for it. In the latter, a lot of people capture data for the sole purpose of measuring the performance of something like how many "hits" their website took in any given time frame. The latter would also imply that asynchronous accumulation of such data might be ok because it's not critical. For the former, I want stuff to stop working if it's not being audited because of the legalities of things continuing to work that aren't being audited.One other part that I think people don't understand as a form of "auditing" are things like "orders". If your invoice detail table can't be accessed as a form of audit for the orders taken, you need to stop the applications. Not all "audit" data appears in tables with the word "audit" in the name. 😉
Heh... as with all else, "It Depends". 😀
That was what I was trying to get across but failed to do so :-S
Nah... you did fine. I totally got your last and agree.
Shifting gears a bit and to be totally honest... if someone can't do logging/auditing in real time without it slowing things down to the point of being noticeable, they should probably consider another line of work because such requirements are only going to get worse as the governments of the world make more and more laws about data and people in IT need to CYA more and more because of it.
--Jeff Moden
May 11, 2015 at 3:38 pm
Jeff Moden (5/11/2015)
Gary Varga (5/11/2015)
Jeff Moden (5/10/2015)
I think a lot of people get "auditing" mixed up with "logging". In the former, it must not fail because there are legal reasons for it. In the latter, a lot of people capture data for the sole purpose of measuring the performance of something like how many "hits" their website took in any given time frame. The latter would also imply that asynchronous accumulation of such data might be ok because it's not critical. For the former, I want stuff to stop working if it's not being audited because of the legalities of things continuing to work that aren't being audited.One other part that I think people don't understand as a form of "auditing" are things like "orders". If your invoice detail table can't be accessed as a form of audit for the orders taken, you need to stop the applications. Not all "audit" data appears in tables with the word "audit" in the name. 😉
Heh... as with all else, "It Depends". 😀
That was what I was trying to get across but failed to do so :-S
Nah... you did fine. I totally got your last and agree.
Shifting gears a bit and to be totally honest... if someone can't do logging/auditing in real time without it slowing things down to the point of being noticeable, they should probably consider another line of work because such requirements are only going to get worse as the governments of the world make more and more laws about data and people in IT need to CYA more and more because of it.
While I don't disagree with the spirit of it, how and where you retain audits are often outside of our control. All things being equal, if the audit requirements exclusively cover the data side of the house and if they allow me to build in the audit controls into the very processing code I am auditing for short-running activities, then yes I might be inclined to make them real-time and synchronous in-line with my code. If on the other hand they cover multiple functional systems, across long-running activities with multiple technology stacks, then you adapt to what you can control, putting in whatever audits controls you can, and logging the rest.
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
May 11, 2015 at 11:02 pm
In that case, I recommend major CYA efforts on the part of the DBAs because that's where the blame will end up if audits fail, audit data is missing or is insufficient, especially if you're in virtually any form of banking, insurance, or finance because of what they're doing with the laws lately.
Since I didn't want to play the CYA game, I stuck my foot in the management door and insisted that I be a part of any decision having to do with auditing/logging of anything and everything in the places where auditing is required and couple where it wasn't. That last one was important because they didn't think that some of the auditing I wanted was important (they actually thought I was a bit paranoid and even silly at first) and it turned out to be a lifesaver for the Dev Manager, the CIO, and the NetOps Director during a "surprise external agency audit" that our lovely Director of Security sprung on us with no warning.
--Jeff Moden
May 12, 2015 at 8:57 am
Jeff Moden (5/11/2015)
In that case, I recommend major CYA efforts on the part of the DBAs because that's where the blame will end up if audits fail, audit data is missing or is insufficient, especially if you're in virtually any form of banking, insurance, or finance because of what they're doing with the laws lately.
Ditto. This has saved me many times in the past, having records of stuff. Even written records of actions carry weight if you get in the habit and have lots of them.
Follow me on Twitter: http://www.twitter.com/way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
May 12, 2015 at 12:28 pm
I get not wanting to be impactful, but then what's the point of auditing?
It's a blame system with proof:hehe:
May 12, 2015 at 7:50 pm
David.Poole (5/12/2015)
I get not wanting to be impactful, but then what's the point of auditing?
It's a blame system with proof:hehe:
mine's a blame avoidance system with proof, but yep - that's the idea :w00t:
----------------------------------------------------------------------------------
Your lack of planning does not constitute an emergency on my part...unless you're my manager...or a director and above...or a really loud-spoken end-user..All right - what was my emergency again?
May 13, 2015 at 8:10 am
Mine's a bit more altruistic than just keeping my butt out of the sling. To make a really long story short, we had a surprise audit and were given very high marks because of some of the auditing/logging I did and some of the inspection points weren't what anyone would have expected but I'd seen such things in the past and I expected them including such oddities as the relationships between the ticketing system and the peer review/code promotion process.
The opposite can be true on companies in the industry our company is in... if they don't like what they see, they give you a certain amount of time to comply (and it's not a long time) and if they still don't like what they see, they can shut you down. That would put a whole lot of people out of work.
When it comes to synchronous logging, don't take any shortcuts. Just do it an make sure it works every time.
--Jeff Moden
Viewing 13 posts - 16 through 27 (of 27 total)
You must be logged in to reply to this topic. Login to reply