Showing posts with label SQL Server Mobile. Show all posts
Showing posts with label SQL Server Mobile. Show all posts

Wednesday, August 06, 2008

The Importance of SqlCeCommand.Parameters

One popular post on this blog discussed the reasoning behind the SqlCeCommand.Parameters functionality in the .NET SQL libraries. Since the original post back almost two years ago (August 14th, 2006), I have used other SQL libraries in other languages including Java and Python. They also provide parameterized queries in addition to the traditional text based queries. Thus, this design is quite common.

Although you might be tempted to write your queries using string.Format and SqlCeCommand.CommandText, it is much better to let the SQL library interpret your parameters automatically. Ilya Tumanov, a member of the .NET Compact Framework Team, commented on my original post with a list of advantages that are worth repeating:

Yes, parameterized query is a preferred way. Benefits are:
  • No need to create bunch of string objects all the time (in your code).
  • No need to convert parameters from binaries to strings (in your code).
  • No need to parse strings to get binaries back (in SQL CE).
  • No locale/format specific issues as binary is format less and locale independent.
  • No need to redo each command from scratch, prepared execution plan can be used with just different parameters.
  • Works for all data types including binary/image.
Performance benefits from using parameters range from significant to really huge.

The reasoning behind this post is to emphasize the red bullet point above. Recently, I have been receiving emails from users of the MyExperience tool who have tried to run the program on devices that are using non-English language versions of Windows Mobile. For the most part, MyExperience uses parameterized queries. I painstakingly went over every query a few years ago and made sure of that. Unfortunately, I missed a spot and this was the cause of the incompatibility.

The offending query looked something like this:

sqlCmd.CommandText = string.Format("SELECT VersionId FROM Versions WHERE VersionName='{0}' AND VersionNumber='{1}' AND VersionModifiedDate='{2}'", vName, vNumber, vDate);

A keen reader may note that this query is region specific because it includes a DateTime object. The String.Format method was formatting the date based on the regional configuration of the device. This seemed to be incompatible with the date format that SQL Server Mobile was expecting. To fix this, I updated this code to read:

sqlCmd.CommandText = "SELECT VersionId FROM Versions WHERE VersionName=@VName AND VersionNumber=@VNumber AND VersionModifiedDate=@VModDate";
sqlCmd.Parameters.Add("@VName", mev.VersionName);
sqlCmd.Parameters.Add("@VNumber", mev.VersionNumber);
sqlCmd.Parameters.Add("@VModDate", mev.ModificationDate);


Thus, I kept the data in binary format which is, as Ilya pointed out, region/locale independent.

Thursday, January 11, 2007

Keeping SqlCeConnection Open and Thread Safety

I've done quite a bit of searching online and it's fairly clear that there is a bit of confusion about how one should use the SqlCeConnection object, particularly with regards to two interrelated issues: (1) whether to keep the SqlCeConnection open and (2) whether the SqlCeConnection is thread safe. For example, some people advocate for keeping one static SqlCeConnection open and shared for the lifetime of an application. However, if the application is multithreaded, can we assume that we can safely share the connection object? I've blogged about this once before here.

I think some of this confusion stems from the differences between the desktop version of SQL Server (SqlConnection) and SQL Server Mobile (SqlCeConnection). For example, in a MSDN forum post, Dave Hayden (a .NET/C# MS MVP) suggests that "For 99% of all applications, the best practice is to open and dispose of database connections right when you need them and not to leave them open for the duration of the application. Open the database connection as late as possible and close/dispose of it as soon as possible." If, like Dave recommends, we always open a local connection within our method calls (and do not access the SQL connection as a state object), we don't really have to worry about the thread safety of the SQL connection object.

Although this may be good advice for accessing SQL Server from the desktop, it contradicts the advice given by Microsoft developer Marcus Perryman in his SQL Mobile post on his blog, "The SqlCeConnection class implements the IDisposable interface because it allocates a number of unmanaged resources, and therefore the code must call Dispose() on the object before it goes out of scope to ensure these resources are cleaned up in a timely manner. Creating and destroying SQL Mobile database connections is an expensive task and so the SqlCeConnection is designed to be a long lived, shared instance across the lifetime of the application.".

Thus, it appears that Dave and Marcus contradict each other. However, Dave's advice was (I believe) given under the assumption that the person was using SQL Server and not SQL Server Mobile. So, it looks like the correct usage of the SqlCeConnection object is to reduce the amount of creating and destroying SQL Mobile database connections (the reasoning behind this is that SQL Server Mobile supposedly does not have connection pooling, see Marcus Perry link above).

So, then what about thread safety? The MSDN documentation is clear. Most objects within the SQL Mobile namespace are not threadsafe. For example, the SqlCeConnection documentation states, "Any public static members of this type are thread safe. Any instance members are not guaranteed to be thread safe." Thus, instance methods like BeginTransaction(), CreateCommand(), etc. are not thread safe. I'm not sure what the detriment/exception would be if this was not followed, but MSDN is clear, SqlCeConnection was not meant to be shared across threads.

OK, so what if you have a multithreaded application? It seems like there are two prevailing methods: (1) Marcus Perry suggests that "for a complex app, ideally the SqlCeConnection instance would be placed in a singleton wrapper class that manages access to the database." or (2) Ginny Caughey (MS MVP) offers the other approach, using a separate SqlCeConnection object for each thread. For the first approach, I would imagine the simplest wrapper possible may look something like the following (I will implement it as a static class rather than a singleton):


static class SqlWrapper
{

private static SqlCeConnection _sqlCeConnection;
private static object _objLock = new object();

static void Open(string connectionString)
{
lock (_objLock)
{
if (_sqlCeConnection != null)
throw new InvalidOperationException("Connection already open.");
_sqlCeConnection = new SqlCeConnection(connectionString);
}
}

static void Close()
{
lock (_objLock)
{
if (_sqlCeConnection != null)
{
_sqlCeConnection.Dispose();
_sqlCeConnection = null;
}
}
}

static int ExecuteNonQuery(SqlCeCommand sqlCommand)
{
lock (_objLock)
{
sqlCommand.Connection = _sqlCeConnection;
return sqlCommand.ExecuteNonQuery();
}
}

static SqlCeDataReader ExecuteReader(SqlCeCommand sqlCommand)
{
lock (_objLock)
{
sqlCommand.Connection = _sqlCeConnection;
return sqlCommand.ExecuteReader();
}
}
}


Does this seem right to people? In this class, we do not expose the underlying SqlCeConnection object, thus commands must be passed in via the wrapper interface to be executed (as they require a reference to a SqlCeConnection).

For the second approach, one connection per thread, the difficulty isn't necessarily in opening 1 connection per thread but knowing when to close those connections. In other words, a SqlCeConnectionManager class might manage access to SqlCeConnections and might serve as a SqlCeConnection factory (with, perhaps, an underlying connection pool). The factory may look at which thread is active (Thread.CurrentThread), check to see if a connection has been allocated for that thread (create one if not) and return the SqlCeConnection. However, the onus would be on the threads themselves to close their connections before exiting (this seems messy to me). I suppose one could spawn a monitoring thread that looked as the status of Threads and, once dead, would either close the respective SqlCeConnection or return the connection the connection pool. However, .NET CF 2.0 does not support the instance property IsAlive so it's not clear how one could do this.

I would imagine that a multithreaded application that launches many short-lived threads that need database access should probably go with the SQL wrapper solution as the overhead of opening/closing connections would introduce a performance hit (even with connection pooling).

An interesting side note: I created a test app on the mobile phone which launched 50 threads on a shared SqlCeConnection and proceeded to read/write random bits of data. No exception was thrown. Of course, the lack of error does not mean that it will always work.

Tuesday, August 22, 2006

Size Limit of SQL Server 2005 Mobile Database

Size limit of SQL Server 2005 Mobile database: 4 Gigabytes.
Size limit of SQL Server Everywhere Edition database: 4 Gigabytes.

(link)

Monday, August 21, 2006

SqlCeParameter Bug?

I made the following post to microsoft.public.sqlserver.ce on August 17th.

The following SqlCeCommand throws an exception. Is this a bug or am I doing something wrong?

SqlCeCommand sqlCmd = sqlConn.CreateCommand();
sqlCmd.CommandText = String.Format("INSERT INTO {0} (UserId, VersionId, {1}, PropertyName, PropertyValue, PropertyValueType) VALUES (@userid, @versionid, @headercolumnval, @propertyname, @propertyvalue, @propertyvaluetype)", strTable, headerName);
String strValue = p.Value.ToString();
sqlCmd.Parameters.Add("@userid", SqlDbType.NVarChar, 255, "UserId").Value = Globals.UserId;
sqlCmd.Parameters.Add("@versionid", SqlDbType.BigInt, 8, "VersionId").Value = Globals.VersionId;
sqlCmd.Parameters.Add("@headercolumnval", SqlDbType.NVarChar, 255, headerName).Value = headerColumnValue;
sqlCmd.Parameters.Add("@propertyname", SqlDbType.NVarChar, 255, "PropertyName").Value = p.Key;
sqlCmd.Parameters.Add("@propertyvalue", SqlDbType.NVarChar, 4000, "PropertyValue").Value = strValue;
sqlCmd.Parameters.Add("@propertyvaluetype", SqlDbType.NVarChar, 255, "PropertyValueType").Value = p.Type.FullName;
sqlCmd.Transaction = sqlTransact;
sqlCmd.Prepare();
sqlCmd.ExecuteNonQuery();
sqlCmd.Dispose();

Throws the following exception at the sqlCmd.ExecuteNonQuery() line:

System.InvalidOperationException was unhandled
Message="@propertyvalue : String truncation: max=255, len=463, value="(removed for posting clarity)"."
StackTrace:
at System.Data.SqlServerCe.SqlCeCommand.FillParameterDataBindings()
at System.Data.SqlServerCe.SqlCeCommand.ExecuteCommandText()
at System.Data.SqlServerCe.SqlCeCommand.ExecuteCommand()
at System.Data.SqlServerCe.SqlCeCommand.ExecuteNonQuery()
at MyExperience.IO.DatabasePopulator.InsertProperty()
at MyExperience.IO.DatabasePopulator.ParseTriggers()
at MyExperience.IO.DatabasePopulator.Populate()
at MyExperience.MyExperienceFramework.Initialize()
at MyExperience.Application.MainForm.OnLoad()
at System.Windows.Forms.Form._SetVisibleNotify()
at System.Windows.Forms.Control.set_Visible()
at OpenNETCF.Windows.Forms.Application2.RunMessageLoop()
at OpenNETCF.Windows.Forms.Application2.Run()
at MyExperience.Application.Program.Main()

However, if I change the line:

sqlCmd.Parameters.Add("@propertyvalue", SqlDbType.NVarChar, 4000, "PropertyValue").Value = strValue;

to

sqlCmd.Parameters.Add("@propertyvalue", SqlDbType.NVarChar).Value = strValue;

no exception is thrown and, better yet, the full amount of data actually makes it into the database. That is, the value is not truncated to the 255 length that Sql somehow thinks exists as a constraint.

Note that my schema for this table looks like:

TableName:
TriggerProperties

Columns:
UserId (PK, nvarchar(255), not null)
VersionId (PK, nvarchar(255), not null)
TriggerName (PK, nvarchar(255), not null)
PropertyName (PK, nvarchar(255), not null)
PropertyValue (nvarchar(4000), not null)
PropertyValueType (nvarchar(255), not null)

I followed this post up with this one on August 18th:

Ah, it looks like SqlCeParameter forces the Size property to 255. That's strange given the fact that the docs say, as previously mentioned, that the nvarchar data type can range from 1 to 4000 characters.

However, bear in mind, that by simply avoiding setting the Size property (either via the constructor or the property itself) allows one to INSERT nvarchar data much larger than 255. In other words, it looks like this "bug" is at the parameter level and not database level.

This is the test code:

for (int i = 0; i <= 500; i+=100){ SqlCeParameter param = new SqlCeParameter("@Test", SqlDbType.NVarChar, i); Debug.WriteLine(i + ": " + param.Size);

//Just in case the property acts differently, set it and print again
param.Size = i;
Debug.WriteLine(i + ": " + param.Size);

}


prints:

0: 0
0: 0
100: 100
100: 100
200: 200
200: 200
300: 255
300: 255
400: 255
400: 255
500: 255
500: 255

Any input from MS (or MVPs) on this? I'm far from being a seasoned C# SQL developer so maybe I'm doing something wrong.

Unfortunately I haven't heard any substantive comments back on usenet from MS or otherwise. Anyone in the blogosphere have an idea?

Update 8/25/2006 @ 11:08PM: On August 21st, I e-mailed the SQL Server Everywhere team via their contact page about this issue. Here's what I said:

Not sure how appropriate it is to contact you about problems I'm experiencing with Sql Mobile 2005, but I thought I'd give this a shot. If this is the wrong venue of communication, just ignore this plead for help.

I'm having issues with the SqlCeParameter class, particularly with regards to how it handles setting the Size property. I can't seem to set the size to anything larger than 255 for the nvarchar datatype.

Please see my usenet post for more details (unfortunately no one with any expertise in this area has responded yet): direct link here


I received a response on August 24th:

Yes Jon you are right that there seems to be some problem with larger than 255 chars. I just tried and could repro it.

I shall come back to you on this after some more digging around to see if there is any work around.


To which I responded:

Thank you so much for the response. I was really hoping I'd hear back from someone at Microsoft to help me with this.

The work around that I've come up with is to simply leave out specifying the length parameter in the SqlCeParameter object. So, for my SQL data columns that are of type nvarchar and length > 255 I do this:

SqlCeParameter param = new SqlCeParameter("@custid", SqlDbType.NVarChar);


Rather than explicitly including a length and column:

SqlCeParameter param = new SqlCeParameter("@custid", SqlDbType.NVarChar, 2000, “CustomerID”);


You’ll note that in the former case your data can be of length > 255 and, critically, it will still make it into the database. In the latter case, however, you incur an exception.

Perhaps my workaround will shed some light on the underlying problem.


On August 25th I received another response:

I found the work around you used works perfectly and some of the internal apps also use it the same way as you mentioned. However, while digging I found that you can not use parameterized query when you have image/ntext columns (Large Object Datatype). Will update you more once I get some clarity.

To which I responded:

Hmm, I also use image data types in my SQL Mobile 2005 database and it appears to be working. In fact, a parameterized query, I believe, is the only way to get binary data into the database. What was the issue you ran into here?

Monday, August 14, 2006

Usage of SqlCeCommand.Parameters

I posted the following to microsoft.public.sqlserver.ce:

I'm curious about the Parameters property in SqlCeCommand. Is this the preferred/best usage of the SqlCeCommand object? I suppose one benefit of using Parameters is the ability to set binary/image data (as those are byte arrays) for insertions/updates. Are there other benefits? I tend to format all of my command statements like the following:

SqlCeCommand cmd = sqlConn.CreateCommand();
cmd.CommandText = String.Format("INSERT INTO {0} (ActionName,
ActionType) VALUES ('{1}', '{2}')", "Actions", actionName, actionType);
cmd.Transaction = sqlTransact;
cmd.ExecuteNonQuery();

instead of

SqlCeCommand cmd = sqlConn.CreateCommand();
cmd.CommandText = "INSERT INTO @tablename (ActionName, ActionType)
VALUES (@actionname, @actiontype)";
cmd.Parameters["@tablename"].Value = "Actions";
cmd.Parameters["@actionname"].Value = actionName;
cmd.Parameters["@actiontype"].Value = actionType;
cmd.Transaction = sqlTransact;
cmd.ExecuteNonQuery();

What's the "best practice?"

Ilya Tumanov, a member of the .NET Compact Framework Team, responded with:

Yes, parameterized query is a preferred way. Benefits are:
  • No need to create bunch of string objects all the time (in your code).
  • No need to convert parameters from binaries to strings (in your code).
  • No need to parse strings to get binaries back (in SQL CE).
  • No locale/format specific issues as binary is format less and locale independent.
  • No need to redo each command from scratch, prepared execution plan can be used with just different parameters.
  • Works for all data types including binary/image.
Performance benefits from using parameters range from significant to really huge.

(link)

Update 08/16/2006 12:57AM: I followed up the above post with:

Thanks so much for this thorough response--it clarified much for me. One follow-up question, what do you mean by "No need to redo each command from scratch, prepared execution plan can be used with just different parameters?" This is most certainly my naivete here with respect to SQL; a short elaboration would probably suffice.

Illya once again responded:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sqlc...

Preparing Commands
To be able to execute a query, the database engine must first parse, compile, and optimize the SQL statement. Often, this work can be done once if the command is to be executed multiple times, potentially saving time. If clients expect to run a query more than once, it is recommended that the command be prepared once, then call Execute multiple times. This should maximize performance by avoiding query recompilation. Commands can be prepared prior to execution by calling ICommandPrepare::Prepare. This is equivalent to compiling the command.

Real life equivalent would be a move to another apartment. Would you rather move your stuff item by item or pack it and move it all at once?



Monday, July 31, 2006

SQL Mobile Size Limit

Ginny Caughey, a Device Application Development MVP, wrote about SQL Mobile size limitations in the microsoft.public.sqlserver.ce newsgroup (direct link here):

The limit for a SQL Mobile database is 4 gb, so unless my math is wrong, you shouldn't have any problems at all with 80,000 records of that size assuming you enough space on the device. I have a production app that handles 65,000 records and on a mobile device seeking to a unique key using the table's index is almost instantaneous. One thing I have found with big databases is that sometimes queries get very slow because the query processor apparently wants more memory for its own work than I have available. I've been able to reduce a 30-minute query to well under a second by switching from using SQL syntax to TableDirect in situations like that.

One other point with large amounts of data - don't try to load all that data into a DataSet. Use SqlCeResultSet and only load the data you need at any given point in your app.

Saturday, July 15, 2006

SqlServer Mobile Edition, In-Proc Database

I just posted the following to microsoft.public.sqlserver.ce:

On Sriram Krishnan's blog he writes, "Sql Mobile is a super-lightweight, in-proc database that runs on devices. The key phrase for me personally is 'in-proc'. This means that there is no IPC involved and as a result, performance is blindingly fast."

This is really the only discussion I could find about in-proc databases via a quick Google search. Can anyone provide me with any more information on what an "in-proc database" is and how it differs from an "out-of-proc?" database. The implication of Sriram's post is that SqlMobile is efficient because it doesn't use interprocess communcation (assuming that's what IPC stands for). Do most databases use IPC?

(direct link)

(Update: 07/16/2006 1:37PM) I received the following response from Ginny Caughey, a Device Application Development MVP.

Some databases such as SQL Server run as services in their own process space, often on a different machine from the apps that consume those services. Others such as SQL Server Everywhere/SQL Mobile are DLLs that run in the same process space as the application that uses them. Not only does this imply that there is no process boundary to impact perforamance, but also when the hosting application ends, so does the database engine so there is no service running to serve as a potential security threat. There are advantages to client/server databases such as SQL Server, but for stand-alone and mobile apps, there are also advantages for an in-process database such as SQL Mobile/Everywhere.

(direct link)

Thursday, June 22, 2006

Sql Server 2005 Mobile

Astonishing as it now seems to me, I had not used SQL before integrating Sql Server 2005 mobile into my current project. That said, I occassionally run into these problems that could only happen to a SQL neophyte. For example, I've found out the hard way that naming your table columns either Trigger, Default or Option is a bad thing and confuses the hell out of SQL (and your development team members). Trigger, Default and Option, as I'm sure most of you know, are reserved words in Sql.

Tuesday, April 11, 2006

Primary Key Efficiency

Not sure if this is relevant to SQL Server Mobile or SQL Server 2005, but still interesting nonetheless. The key takeaway seems to be that the primary key should be single column if possible and an int datatype (link).

Also, from link, "A primary key is an attribute (or combination of attributes) that uniquely identify each instance of an entity. A primary key cannot be null and the value assigned to a primary key should not change over time. A primary key also needs to be efficient. For example, a primary key that is associated with an INTEGER datatype will be more efficient than one that is associated with a CHAR datatype. Primary keys should also be non-intelligent; that is, their values should be assigned arbitrarily without any hidden meaning. Sometimes none of the attributes of an entity are sufficient to meet the criteria of an effective primary key. In this case the database designer is best served by creating an "artificial" primary key."

Monday, April 10, 2006

Multiple Columns as Primary Key?

I was questioning whether my database design was a good one. This is my first project using SQL (or any database for that matter) and I wondered whether using multiple columns as a primary key was "good design" or, alternatively, if I should just create a dummy column in my tables and use the IDENTITY property (so that they are auto-numbered and guaranteed unique). I guess this is a hotly debated issue. I found quite a few discussions on the web about it--with people arguing adamently in both directions.

Here are a few links.

1. "Primary Key" (Google Groups).
2. "Design: multiple columns for primary key" (Google Groups)
3. "SQL Server Performance Questions & Answers" (in favor of IDENTITY columns)

Adding Foreign Keys to .SDF via VS2005

From msdn forums:

"You can't add a foreign key to a SQL Mobile database from within VS2005 - you cannot do it graphically nor by using the query designer (where it would be tempting to enter an ALTER TABLE .. ADD FOREIGN KEY statement).
In the absence of SQL Server 2005 Management Studio in your situation, you will need to run the ALTER TABLE command either from your program code or inside of Query Analyzer 3.0 on device."

(link)

Sunday, April 09, 2006

SqlCeConnection Keep Connection Open?

Consensus seems to point to keeping one instance of SqlCeConnection open for the duration of executation for performance reasons.
  1. SqlCeConnection guidelines (MSDN Forums)
  2. Performance with SqlCeConnection (Usenet)
  3. Data Architecture & Pooling (Usenet) <- this is for SqlServerCe 2003
Marcus Perry in his MSDN blog notes:
Creating and destroying SQL Mobile database connections is an expensive task and so the SqlCeConnection is designed to be a long lived, shared instance across the lifetime of the application. For a complex app, ideally the SqlCeConnection instance would be placed in a singleton wrapper class that manages access to the database.

Though this information comes from a Microsoft employee, it still seems contentious. Dave Hayden, a MS MVP, said in this MSDN forum post:
For 99% of all applications, the best practice is to open and dispose of database connections right when you need them and not to leave them open for the duration of the application. Open the database connection as late as possible and close/dispose of it as soon as possible.

Nicholas Paldino (a .NET/C# MVP) seems to agree with Dave:

You shouldn't keep a connection open and lying around. The SqlConnection class uses a connection pool in the background, and keeping your connection open is actually going to degrade your performance. You will want to create a new connection, open it, use it, and then close it for every operation that you do (or group of operations). You shouldn't be holding the connection open across operations. You will also notice MUCH better performance when you do this. The same thing holds with the command object. Don't use the same one. Use new ones. If you want to use a command object over and over again, you can, but use it for the same parameterized command. Don't keep changing the command string.

Both Dave and Nicholas are basing their advice on SQL Server's behavior (not SQL Server Mobile). SQL Server supports connection pooling, SQL Server Mobile does not. Again, from MS Employee Marcus Perry (link),
There is no connection pool used for local database access for SQL Mobile.

If only one SqlCeConnection is used, note that it is not thread safe. Thus, a multithread application must provide mechanisms to safely access the SQL database from multiple threads. There seem to be two approaches: one is to make the singleton wrapper class (mentioned above) thread safe and the other is to give each thread access to its own SqlCeConnection object. See these MSDN forum posts (here and here). Microsoft has yet to endorse a specific approach as a "best practice" (a discussion about this can be found here).