Author |
Topic |
theboyholty
Posting Yak Master
226 Posts |
Posted - 2012-03-22 : 07:24:54
|
I almost never hear anyone say anything good about cursors, rather people refer to 'set based' programming as a preferred alternative. Truth is, I don't really know what is meant by set-based operations (although I may find that it is what I have been using all this time).I would personally use a WHILE loop to iterate through a dataset because I find the syntax easy to implement and performance seems to be ok generally speaking (although I'm using fairly small datasets), but would be interested to hear how other experienced SQL Developers feel on the subject.Cheers.---------------------------------------------------------------------------------http://www.mannyroadend.co.uk A Bury FC supporters website and forum |
|
GilaMonster
Master Smack Fu Yak Hacker
4507 Posts |
Posted - 2012-03-22 : 08:49:10
|
A cursor is a while loop by another name, nothing more. Any form of looping is inefficient, whether it be cursor, while loop or other mechanism.Set based programming is when you process a set of data in a single operation instead of looping over a set of rows and processing them one by one.--Gail ShawSQL Server MVP |
 |
|
khtan
In (Som, Ni, Yak)
17689 Posts |
|
theboyholty
Posting Yak Master
226 Posts |
Posted - 2012-03-23 : 05:46:47
|
That's a really useful explanation, but I still can't see a better way of doing what I need to do, through set based SQL.I generate tables of up to a few thousand rows of data, and in each row we've built a URL for posting data to our clients' external systems. This is a stripped down version of the code.DECLARE @c INT = 1, @PostURL VARCHAR(1000)WHILE @c <= (SELECT COUNT(*) FROM FM_Magazine_WebSelection)BEGIN SET @PostURL = (SELECT PostURL FROM FM_Magazine_WebSelection WHERE RowNumber = @c) EXEC usp_CLR_PostURL @PostURL SET @c = @c + 1END As you can see, it loops through each row of the dataset, and individually fires the CLR proc for each URL string generated. I'd like to see if this can be done without the loop if possible.---------------------------------------------------------------------------------http://www.mannyroadend.co.uk A Bury FC supporters website and forum |
 |
|
GilaMonster
Master Smack Fu Yak Hacker
4507 Posts |
Posted - 2012-03-23 : 07:06:10
|
What does the CLR proc do? Turning that set-based will involve changing the procedure somehow, hence will need to see the code.It's worth noting that CLR procs are always iterative, that's the nature of .Net, and there are some things (like sending email) that can't be done set-based. Those are exceptions and should not be the norm.--Gail ShawSQL Server MVP |
 |
|
theboyholty
Posting Yak Master
226 Posts |
Posted - 2012-03-26 : 11:16:50
|
Thanks for having a look at this. Here's the vb.net code which basically posts a URL to the web to send data to an associate's database via the web.An example of the parameter that might be passed to it is as follows :htp://leadintelrg.com/leads?cid=45843&siteid=340&title=Mr&first_name=John&last_name=Smith&email_address=js@test.com&telephone_number=01234567890(by the way it does really begin with 'http://' but when I submitted my response it formatted it weirdly)Imports SystemImports System.DataImports System.Data.SqlClientImports System.Data.SqlTypesImports Microsoft.SqlServer.ServerImports System.Data.SqlImports System.IOImports System.NetImports System.TextImports System.WebImports System.CollectionsImports System.ComponentModelImports System.Text.RegularExpressionsImports System.DiagnosticsImports System.Web.ServicesPartial Public Class StoredProcedures <Microsoft.SqlServer.Server.SqlProcedure()> _ Public Shared Sub usp_CLR_PostURL(ByVal InStr As String) Dim request As HttpWebRequest Dim response As HttpWebResponse = Nothing Dim reader As StreamReader Dim URLString As String Dim address As Uri Dim returnstring As String URLString = InStr address = New Uri(URLString) request = DirectCast(WebRequest.Create(address), HttpWebRequest) request.Timeout = 1000000 response = DirectCast(request.GetResponse(), HttpWebResponse) reader = New StreamReader(response.GetResponseStream()) returnstring = reader.ReadToEnd Using connection As New SqlConnection("context connection=true") connection.Open() Dim command As New SqlCommand("SELECT '""" & Replace(returnstring, Chr(39), "") & """'", connection) SqlContext.Pipe.ExecuteAndSend(command) End Using End SubEnd Class Thanks.---------------------------------------------------------------------------------http://www.mannyroadend.co.uk A Bury FC supporters website and forum |
 |
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2012-03-26 : 12:15:26
|
I'm not a .Net programmer, but it looks like you are passing a URL to the CLR procedure, requesting the the response from that URL (getting the web page), and then returning the contents of that page, is that correct?One thing I can say for certain, there's no need to open a SQLContext.Pipe just to do a simple text replace, and it's definitely NOT a good way to do that. |
 |
|
Vinnie881
Master Smack Fu Yak Hacker
1231 Posts |
Posted - 2012-03-26 : 12:56:22
|
The proper way to accomplish the task in my opinion is the following.1:vb.netInitiate a Procedure call to return all URL's (ONE SINGLE CALL)2:SQL write the procedure to return all informationVb.net3:Return ALL data to VB.NET and build a XML file (SOAP/WEBSERVICE/ETC)4:POST ENTIRE XML file with one request/response and PARSE appropriately, if this is not a option due to you do nothave proper access to the host computer, then open 1 web request and loop through the posts, so you are not opening and closing web connections each time. Success is 10% Intelligence, 70% Determination, and 22% Stupidity.\_/ _/ _/\_/ _/\_/ _/ _/- 881 |
 |
|
yosiasz
Master Smack Fu Yak Hacker
1635 Posts |
Posted - 2012-03-26 : 16:32:19
|
cursor loops have been good to me when used with set based processingThis is what I have used successfullydo you have a key column on your source data?1. create temp dbcreate TABLE [dbo].migration_temp(KeyField int not null)GO2. Create a processing table that holds all rows that you want to move based on a key fieldcreate table dbo.Process(keyFieldId int NOT NULL, isProcessed bit NOT NULL default((0)))3. Dump into Process all unique keyField values insert into dbo.Process(keyFieldID) SELECT keyFieldID FROm SourceTableView4.dump batch rows into temp table WHILE (SELECT COUNT(*) FROM dbo.Process WHERE isProcessed = 0) > 0 INSERT INTO dbo.migration_temp(KeyField) SELECT TOP 100 KeyFieldID FROM dbo.Process with (nolock) WHERE isProcessed = 06. Batch process your code7. Update process table UPDATE tgt SET tgt.isProcessed = 1 FROM dbo.Process tgt with (nolock) inner join dbo.migration_temp src on src.KeyFieldID= tgt.KeyFieldID <><><><><><><><><><><><><><><><><>If you don't have the passion to help people, you have no passion |
 |
|
Vinnie881
Master Smack Fu Yak Hacker
1231 Posts |
Posted - 2012-03-26 : 21:34:47
|
yosiasz,I am not sure if your approach will add any performance benefit to what he already has, also in just a note, in your while loop it may be best to only check the count one time and set it to a variable and increase a counter in the loop to avoid needing to run that (Select Count(*)... ) each time.I would strongly suggest to use a method to return all data via one db call and post via a single webrequest. I have done numerous software apps prior using .net code where the data is derived from via databases, and for me it has provided drastic improvements over individually processing each record re-initiating all connections and everything else every time.There are situations where this can not be accomplished if the receiving computer is something that you do not have control over to modify to handle the post, but even in that scenario you can query all the data you need in one call, open a web connection to the posting site and quickly post all data in a loop on the .net end.It is just my opinion, but the current approach you have illustrated I would personally not use. Success is 10% Intelligence, 70% Determination, and 22% Stupidity.\_/ _/ _/\_/ _/\_/ _/ _/- 881 |
 |
|
yosiasz
Master Smack Fu Yak Hacker
1635 Posts |
Posted - 2012-03-27 : 01:16:32
|
Vinnie I agree with you what I provided might not add any performance approach. But it depends what you are trying to accomplish. there is also the option you mention which is one call that can return xml/json data that can be processed in the .net side of things. that is by far the best approach for his situation<><><><><><><><><><><><><><><><><>If you don't have the passion to help people, you have no passion |
 |
|
|