Completely Empty the SharePoint Recycle Bin

If you have LOTS of stuff in your SharePoint Recycle Bin and want to empty it all, here’s a quick tip:

1. Navigate to the recycle bin page (e.g. http://localhost/_layouts/recyclebin.aspx)

2. In the address bar, type “javascript:emptyItems()” (minus the quotes)

3. Press enter. You will be asked if you really want to empty the recycle bin.

4. Click OK to empty it; click Cancel to leave it alone.

It’s much faster to empty it this way if you have multiple pages of “stuff” in your recycle bin.

ReaderWriterLock FTW!

Recently I was encountering an issue on a SharePoint site where some site properties were getting cached in the HttpContext.Current.Cache.  The logic went roughly like this:

1. Read values from a SharePoint list
2. Put the values into the cache

I'm oversimplifying, of course, but those are the important steps as they relate to this blog post.
I had implemented proper locking, or so I thought, by using the C# lock statement.  The pattern I learned long ago when dealing with creating a shared cache that multiple threads can access is to do something like this:

1. Try accessing the object.
2. If you get it, great! Go forth and use it.
3. If you don't get it, try obtaining a lock (i.e. you're about to enter a critical section).
4. Once you obtain the lock, try accessing the object again, just in case another thread beat you to it.
5. If you get it, great! Release the lock, go forth and use it.
6. If you still don't get it, create it, then put it in the cache.
7. Exit the critical section.

This pattern prevents unnecessary locking if the object already exists in the cache.  It also prevents the object from being created a second time.
As it turns out, the HttpContext.Current.Cache object isn't entirely thread-safe.  Even though I was locking and such, I was unable to retrieve the site properties I was creating.  To the second thread, it appeared that the properties were not in the cache, and the logic would get invoked to create it again.
That's when I did some research and discovered the System.Threading.ReaderWriterLock class.
This gem allows multiple threads to access an object, but only allows one thread at a time to write to that object.  The object has both a reader lock and a writer lock.  Many reader locks are granted as long as there are no writer locks granted (or pending).  If there is a writer lock that is being requested, it blocks until all the reader locks are released.
Another cool feature is that, once you have a reader lock, you can call "UpgradeToWriterLock" if you discover you need to write to the object.  Once you're done, you then "DowngradeFromWriterLock" and then "ReleaseReaderLock".  Or you can simply call "ReleaseLock", I believe, to release all locks obtained.
Best practice for using this object is to wrap it in a try…finally block of code. In the "try" part, acquire the lock and then do your stuff. In the "finally" part, release the lock.  In this manner you ensure that you always release the locks you acquire, and the system never gets "out of balance".
Here's how I used it to solve my problem:

1. Get a reader lock (will block only if another thread is writing)
2. Try to get the properties
3. If obtained, exit and use the properties (the finally kicks in to release the lock)
4. If not obtained, upgrade to a writer lock (will block if any other reader or writer locks are being held).
5. Try to get the properties again (nested try block)
6. If obtained, exit and use the properties (the finally kicks in to downgrade the lock, and then the outer finally kicks in to release the reader lock)
7. If not obtained, create it and return it (the finally blocks kick in as described in step 6)

It's more complex than using a simple lock statement, but I found that it was necessary in this case.
One final implementation note: the ReaderWriterLock object was stored in a "private static readonly" member variable so that it was accessible from any method that needed it, and so that all threads were accessing the same object.

[Update: adding to CodeProject]

Don't Forget to "DoEvents"

I'm writing a reporting package which uses Microsoft Access and Microsoft Excel to gather and chart data collected from a set of custom SharePoint lists.  I have some VBA code which does some calculations on all of the list items before the reports are generated to ensure that the data is the most up-to-date that it can be.

The VBA code lives in Access.  It basically does a lookup on each row in one of my lists and sets values in another list.  The business rules for this update had enough conditions on it that I couldn't implement it in one complex Update statement, thus the looping.

I also discovered that you can very easily add a progress bar to the Access status bar by calling "SysCmd acSysCmdInitMeter, 'message', maximum value" to initialize it, and "SysCmd acSysCmdUpdateMeter, currentValue" to increment the counter.  (To reset the status bar when I'm done, I call "SysCmd acSysCmdRemoveMeter".)  So I added this bit of functionality to the VBA code as a bonus for my users to give them some feedback.  After all, some of the lists I'm working with have a few thousand items in them (don't get me started about best practices stating that 2000 items is the workable maximum—I've already fought that battle and lost!) and it can take some time for the VBA code to complete.

Well wouldn't you know it, but the update code continually ran, and the thread blocked on the SQL command, so the progress bar just sat there, no progressing.  Furthermore, the whole UI appeared to be locked up while the update was running.  I couldn't even pause execution of the VBA code because the IDE wouldn't recognize my mouse clicks on the pause button.

And then I remembered something from my old VB6 days: the DoEvents method.

The DoEvents method relinquishes control of the processor by the VBA engine to allow other processes to do things. Or more accurately in my case, other threads. It was more important to call it when a background thread was doing heavy processing and you wanted to allow the UI thread a chance to handle the mouse clicks, keyboard presses, and other events which had queued up since the last time it got a time slice from the operating system.

I remembered one main warning about using it, though:

• You don't want to call it too often, since an expensive context switch most likely would take place

So I decided to see if it worked in VBA by adding this block of code:

If currentRow Mod 10 = 0 Then
    DoEvents
End If

In this block of code, currentRow is an integer which gets incremented each time through the loop. It is the variable that I pass to the method which updates the progress bar. You can see that I don't call DoEvents each time through the loop—I want the main processing code to actually finish in a reasonable amount of time, after all!

The IDE didn't complain about the method call, so it at least recognized it as valid VBA code.  So far, so good. But will it accomplish what I want it to?

The short answer: YES!

So now, in the latest version of the code, the DoEvents method is called every 10 rows.  This allows the UI to get refreshed, so I can now pause execution and scroll through the source code (albeit in bursts).

But more importantly, the progress bar now shows progress as the code runs!

DoEvents FTW!

SharePoint Exception Occurred 0x80020009 (DISP_E_EXCEPTION) – UPDATE

A colleague pointed me to the following article: Description of the Windows SharePoint Services 3.0 Cumulative Update Server Hotfix Package (Sts.msp): June 30, 2009.  In it, it makes reference to the DISP_E_EXCEPTION error.  Specifically, it states:

SPList.GetItems(SPQuery) fails when the item count reaches the threshold value 2000, and you will receive the following error message:

Exception occurred. (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))

Presumably, if you have a custom query on a list, and that query returns more than 2000 items (which is the case on the pages where I'm encountering this error), you'll get this exception.  The fix for this, supposedly, is in the June 30, 2009 hotfix package.

I'm in the process of requesting that this hotfix be applied to our SharePoint server.  I'll post again once that happens and let you all know whether it fixes this problem as advertised.

While searching for possible causes for this exception, the blog posts I found mentioned improperly-configured load balancers and other maladies, but not this 2000-item limitation nor the hotfix.  Hopefully this post will help someone else out there who is having a similar problem on their system.  Please comment if that person is you!

Best Practices for Building SharePoint Solutions

A friend forwarded this link to me: 10 Best Practices For Building SharePoint Solutions.  Most are fairly obvious, but sometimes it's good to state the obvious things, lest we forget them.

Performing Outer Joins on a SharePoint List from Access

I'm currently creating reports from the custom SharePoint lists I have been working on over the past few months.  The way I'm accessing the data in my SharePoint lists is through Access 2007, and then I have an Excel 2007 spreadsheet which uses the Access database to get the data. In this manner, I can use the power of Access' database engine to perform queries against my lists, and then just import the data to Excel.  It sounds cumbersome, and perhaps I'm showing some of my ignorance of Access and Excel development, but it runs much faster than the earlier version of the report which uses VBA only to generate similar reports.

That being said, I ran into an issue with one query.  I have a list which tracks FCC filings.  There can be a major filing and a minor filing, both tied to the same physical location.  So there can be one or two filings (items) per location stored in this list.  I needed to report on whether the filings were "complete".  They are considered complete if the major filing is complete, or if the minor filing is complete in the case where a location does not have a major filing.

My SQL statement was structured like this:

SELECT * FROM
(
    SELECT <<completed major filings>>
    FROM Filings
    WHERE Filings.Type = 'Major'
UNION
    SELECT <<completed minor filings>>
    FROM Filings Minor
    LEFT JOIN Filings Major ON Minor.Location = Major.Location
    WHERE Minor.Type = 'Minor'
    AND Major.Type = 'Major'
)

However, this didn't work.  The second part of the query kept behaving like an inner join.  I did some proofs-of-concept to discover that, yes, outer joins do work in Access on actual tables within Access, but they just weren't working on linked tables where the source is SharePoint.

What to do?

After beating my head against the wall for a while, and possibly pulling some hair out, I finally had the bright idea to break things into smaller chunks.  Thinking that Access had a problem joining a linked table with itself, I created an Access query which only returned items representing minor filings, and another Access query which only returned items representing major filings.  Then I removed the where clauses from my original SQL statement above and replaced direct access to the linked table with the queries I had created.

It worked beautifully!

The new SQL Statement looks like this:

SELECT * FROM
(
    SELECT <<completed major filings>>
    FROM AllMajorFilings
UNION
    SELECT <<completed minor filings>>
    FROM AllMinorFilings Minor
    LEFT JOIN AllMajorFilings Major ON Minor.Location = Major.Location
)

Lesson learned: when working with Access, break things into smaller components to avoid confusing Access.

I can't wait to get back to "normal" C# development!

Strange Column Names

In SharePoint 2007, you can have a column named "!".

I'm sure there are other characters you can use, as well, but this one surprised me. I suppose it is because I'm a C# developer…the "!" (or "bang") is an operator, not an identifier.

But in SharePoint (at least the 2007 version) it is a valid column name.

[Update: It appears that the column named "!" appears in SharePoint Designer as "_". I haven't tried including this column in any of my workflows, so I'm not sure how it would behave. If you have the opportunity to do so, please leave a note letting me know how it works for you. -sj]

SharePoint “Lookup” Columns

As you may know, you can include a “Lookup” column in your SharePoint 2007 lists which, in effect, creates a reference to another SharePoint list.  This can be used, for example, if you have an address list and want to ensure that your states or provinces are all spelled correctly—just create a “StatesProvinces” list, fill it with the properly-named states and provinces, then add a reference to it in your address list. The user experience is a combo box containing the items in the “StatesProvinces” list, and they can select the one they want without having to type it themselves. [Note that you can configure SharePoint to allow multiple items from the referenced list to be selected, but that’s not what I’m focusing on in this post.]

In order to add a lookup column to your list, you first must have created the other list that you are going to reference.  You don’t actually need data in it, yet, but it does need to be created.  Fair enough.

Next, after choosing the list, you must choose which column in that other list you want to display to the user.  So in this example, you may want to have the “Title” column display to the user.

Once you save everything and add data to the “StatesProvinces” list, you have a workable solution.

But what if you want to have a workflow run against this address list which has the reference to the “StatesProvinces” list?  This is where you need to know a little bit about the inner workings of lookup columns in SharePoint.

If you know anything about database development, you probably know about the concept of a “foreign key”. In short, a foreign key resides in a table that is related to another table.  Let’s call the first table the “Child” table, and the second table the “Parent” table.  So the Child table would hold the foreign key to the Parent table.

What does a foreign key look like? Simple: it is the primary key of the Parent. [A primary key uniquely identifies a row in the database table.] It is usually a numeric type.

The same concept applies to a lookup column in SharePoint, but the execution is slightly different. You see, the actual value stored in the lookup column is the ID of the referenced list, but what is displayed to the user is the configured column from that list.

Back to my workflow scenario: if you want to manipulate lookup columns from a SharePoint workflow, and I’m talking specifically about using SharePoint Designer to create the workflows, you need to keep the foreign key concept in mind. As a test of this, do the following:

1. Create a workflow and attach it to a SharePoint list which contains a lookup column.
2. Create two variables in the workflow, one of type “String” and one of type “List Item ID”.
3. Set both variables to the value of the lookup column

If you examine the contents of each variable (by logging it to the workflow history) you will see that the String variable contains the value of the configured column to display (in our example, the “Title” column), and the List Item ID variable would have a number (which corresponds to the ID column of the referenced list).

If you are going to change the value in the Lookup column via a workflow, you must set its List Item ID (number), not the displayed value(text).  You will wonder why your workflow isn’t working correctly if you don’t follow this advice, especially if you are new to creating SharePoint workflows using SharePoint Designer.

I hope I have helped someone out there with this advice.

SharePoint Exception Occurred 0x80020009 (DISP_E_EXCEPTION)

My, oh my, will wonders never cease?

Have I mentioned how much I love SharePoint recently? Let me count the ways. And believe me, there are not 0x80020009 ways. In fact, I can count the ways on one hand. Sometimes no hands! :-)

For some reason, one of my SharePoint lists will not display in the customized Datasheet view that I created. The others custom Datasheet views on my other lists are working perfectly.

So I “Googled with Bing” for a solution and found that, lo and behold, I am not alone in the universe. Many, many more people share my woes with this error.

Some people mention that all that needs to happen is an IIS reset. Well that won’t work for me, since I’m at a very large corporation and I don’t know who the admins are nor do I even know where they are.

Some people mentioned problems with their custom web parts. No, no web parts here. The SharePoint site I’m working on does not allow custom code to be deployed to it. All I have to work with is SharePoint Designer.

Other people mentioned that it was the load balancer which was not configured properly.

Are you kidding me? One error has so many different causes? Do you think someone on the SharePoint team would think to provide us, the lowly end-users, with more of an explanation of where to look for the problem? (Before you mention reading log files or anything similar, keep in mind what I mentioned above: I don’t have access to the servers. I don’t even know where they are!)

Nice. Even though I initially found comfort in knowing that other people have encountered this error and have, in fact, found solutions, it seems that I may be all alone in this particular case.

I tried re-creating the view, just in case something mucked it up. You know, like those stray gamma rays that knock various electrons out of their paths inside your PC’s CPU which cause programs to run amuck. As you probably guessed, it didn’t work.

In fact, as of this writing, I still do not have a solution for this problem. I apologize if you have read this far expecting a solution. Or perhaps my prose caused you to skip ahead, which is good for you. The prose was good for me—it allowed me to vent some steam.

[I ended up deleting all items from the list, and the view started working. Or at least I no longer get the exception. Of course, there were no items in the list for it to display! Once I loaded it with data again, I started getting the same error. Other similar views don't give me this error. Perhaps it's a data issue.]

How Can You Tell if You Found a Valid List ID?

I’m currently working with several different custom SharePoint lists.  Some of the lists have references to other lists, via a “Lookup” column.  For example, a book might have a reference back to the library it came from.

Sometimes I only know the name of the library, and I want to be able to set a reference to it on the book’s list.  So I retrieve the library’s ID using the name of the library.  SharePoint Designer dutifully warns me that I’m not looking up the value using a unique ID, and that it will just give me the first one it finds.  I’m OK with that, since I control the names and I know that they’re all unique.

But I’m only human, and sometimes I don’t have a library name that exists in my library list.

In those cases, when I log the contents of the list ID variable into which I placed the data, it shows me “0”.  [That’s the number zero, not to be confused with a lower-case “o”.  I’m considering changing the font for this blog… -sj]

As it turns out, this is the way you can tell if you have a valid list ID.  Valid IDs start at 1, I presume, and invalid IDs are all zero.

[It’s easy to tell if you have a bad reference to another list when you retrieve string values, since they show up in log messages as “????”.]

Workflow Comparisons Aren’t Always Accurate

I have a SharePoint workflow, developed with SharePoint Designer, which is supposed to compare values in the list it is attached to against values in another list, and copy the other list’s values if they are different.  I kept noticing, via a workflow history log message, that the workflow was copying values every time it ran, even though in another log message for the same workflow instance it showed me that all the values were the same.

As it turned out, I was comparing a string value to a list ID.  Even though they both printed in the log message exactly the same (and so were “equal” in my eyes), SharePoint saw them as different values.

The fix was to copy the list ID value to a string variable, and then compare the two values as strings.

[As a side note, what made this worse was that I had a second workflow which would run when anything was changed on the list.  It would then update a value in the list, which would then cause the buggy workflow described above to run again.  For the astute readers, this meant that the two workflows attached to the same list would cause each other to run infinitely.  It pretty much brought the site to a halt!]

Creating a Custom SharePoint Permission Level

In yesterday’s post, I mentioned custom SharePoint permission levels.  Here is how to create one.

From the landing page of your site (default.aspx), click on “People and Groups”, then click on “Site Permissions” under “All People”. Click on the “Settings” menu, then select “Permission Levels”.

Now click “Add a Permission Level”.  Give your custom permission level a name, an optional description (I find it useful to describe the permissions that this permission level grants, and possibly which assumed permissions that this level does not grant).

Now is the fun part.  You get to choose which permissions to grant, and which to exclude, for this permission level.  There are many different permissions you can choose from.  They are described in this Office Online page.

SharePoint Permission Tip

If you use the “Collect Information from User” action in a SharePoint Designer workflow, and you have your default site permissions set only to “read” for the user you’re planning on collecting information from, you probably won’t get very far.

You’ll want to grant at least “Edit” permission to the user (or group that the user is in—that would be a better practice) on the Task list.  This allows the user to access the task that gets assigned to her so that she can complete her work.

Here’s how to accomplish this:

1. Click on the “Tasks” list on the Quick Launch pane if it is available.  Otherwise, click on “Lists” first, and then click on the “Tasks” list.
2. Click on “Settings | List Settings”
3. Click on “Permissions for this list”
4. Click on “Actions | Edit Permissions”
5. Click “OK” on the resulting dialog box (“You are about to create unique permissions for this list…”)

Now you can modify the permissions for the users and groups listed.  If you don’t see the users or groups you want to configure for the Tasks list, you can add them here.

I must note that I created more granular permission levels on the site I’m currently working on, so I can choose to grant “Read” access or “Edit” access individually.  If you only have the out of the box permission levels, you’ll want to ensure that your users have at least “Contribute” access to the list. Of course, if you only have the out-of-the-box permission levels and you haven’t created custom levels then you wouldn’t have the problem I had which prompted me to write this post.

I’ll cover creating custom permission levels in a future post.

del.icio.us Tags: sharepoint,tip,permission level,tasks

SharePoint? No, SwearPoint!

[Cross posted from my personal blog. -sj]

del.icio.us Tags: sharepoint,development,rant

I’ve been working on a SharePoint project for work for the past few months, and I’m getting close to the “go-live” date.  Actually, it’ll go live whenever I’m “done” (whenever that will be…).

But that’s beside the point.

The point is that I’m becoming less and less impressed with SharePoint as a development platform.  And I didn’t start out with a great love for it in the first place.

Perhaps it’s because I’m constrained to only using SharePoint Designer.  No .NET code allowed on the site I’m working on.  That bums me out right there, since I consider myself a great .NET developer.

Most of my work has revolved around customizing SharePoint lists, and using SharePoint Designer to create custom workflows to support the operations of the team I’m developing this for.  Not necessarily hard, but the requirements have been challenging to implement, to put it nicely.

One main requirement, as it turns out, isn’t really possible to implement fully in SharePoint, at least in a nice way.  There are two groups of people who are responsible for different items (a.k.a. “fields” or “columns”) in a list.  One group, let’s call them “Group A”,  doesn’t want the other group (“Group B”) to be able to modify Group A’s fields.

Now let me stop right there!  I understand that SharePoint is a “collaboration” tool, so preventing users from collaborating just doesn’t jibe with the core of SharePoint.  That’s pretty much the crux of the problem.

If it were possible to set permissions on the field level, this blog entry would not be needed.  But it is not possible to restrict access to only certain fields in a list to a certain group of people, so I had to think of ways to enable this functionality, and the way I came up with is sub-par at best.

I ended up creating 3 lists: one for Group A, one for Group B, and one which shadows Group B’s list.  Then there are workflows which copy Group A’s fields in Group A’s list to Group B’s list, and Group B’s fields in Group B’s list to Group A’s list.  The “shadow copy” list is there to prevent the workflows from running in an infinite loop—the items are only copied back to Group A’s list if they are different between Group B’s list and the shadow copy list.

As it turns out, this is not without problems, since if someone in Group A modifies one of Group B’s fields in Group A’s list (did you follow that?) then it will remain out of sync with Group B’s list, since Group B’s fields aren’t copied from Group A’s list to Group B’s list.

As it turned out, thankfully, this wasn’t as big of an issue for this particular solution (whew!).

It was my intent to set permissions on the lists such that people in Group A only had read access to Group B’s list, and vice versa.  However, as it turns out, the workflows will fail to copy data since they run as the user who initiates the change to the list.  So when a Group A person changes an item in Group A’s list, the workflow tries to copy that change to Group B’s list but can’t because Group A only has read-only access to Group B’s list.

So I thought I could change the permission to allow Group A to edit Group B’s list, but then just set the “Hide from browsers” property on Group B’s list so that Group A couldn’t find it when browsing the site.  So instead of preventing a modification of Group B’s list through the use of SharePoint Permissions, I would be preventing the modification of Group B’s list by making it very hard to find the list.  Problem solved!

Not.

Once again SwearPoint, er, SharePoint worked against me, and here’s how:

If you set the “Hide from browsers” property on the list, it does, in fact, hide the list from site when you click on the “Lists” or “All Site Content” links in SharePoint.  HOWEVER, it also hides the list from workflows!

[UPDATE: The list’s name is the only thing hidden from the workflow.  The error I was seeing which led me to the previous conclusion was due to a user-error (and, no, I wasn’t the user!).  The workflow continues to operate successfully if the referenced list is hidden.]

So my final solution still has three lists, one for each group and a third to ensure that the workflows eventually stop, and each group can see the other group’s lists.  Not at all what I set out to accomplish!

It’ll just have to be a training issue to keep each group out of the other’s list.

So much for collaboration (at least in this project).